Stefan Profanter's questions

How can I force postfix to send out mails sequentially and not simultaneously? It should open only one SMTP connection at a time.

It should only affect outgoing SMTP connections. Incoming connections can still be handled simultaneously.

Why?

My VPS Provider (contabo.de) had the very funny Idea to tie the hands of spammers by limiting the number of simultaneous SMTP connections to 1. I'm not sending SPAM, but a script is sending important order mails to 3 recipients at the same time once a day. This is then blocked with Connection refused in postfix and the mails are deferred. Sometimes it takes over two hours to send the mails out, because at every try the simultaneous connections are blocked.

I have a Windows Server 2008 R2 (Hostname server.domain.local) with DNS and DC service installed and the server has two network cards:

• Card 1: 192.168.10.1
• Card 2: 192.168.20.1

If I open the DNS service properties, I can see two A records:

192.168.10.1 A server.domain.local
192.168.20.1 A server.domain.local

If I manually delete the second entry (because I don't want it to be in the DNS), it always gets recreated the next full hour.

How can I disable this automatic update, so that there is permanently only the first DNS record within the list?

I already tried the steps mentioned in https://support.microsoft.com/en-us/kb/2023004 but this didn't solve the issue.

I have the problem that my apache2 doesn't start at boot. After debugging I found out that a webpage init script tried to connect to MySQL which isn't running at that time.

My OS is Ubuntu Server 10.04.4

apache2 boot is set up using update-rc.d apache2 defaults 21 which creates the scripts in /etc/rcX:

root@ser:~# find /etc/rc* -name *apache*
/etc/rc0.d/K21apache2
/etc/rc1.d/K21apache2
/etc/rc2.d/S21apache2
/etc/rc3.d/S21apache2
/etc/rc4.d/S21apache2
/etc/rc5.d/S21apache2
/etc/rc6.d/K21apache2

and calls /etc/init.d/apache2

mysql is getting started by Ubuntu's upstart:

root@ser:~# ls /etc/init | grep mysql
mysql.conf

How can I force apache2 to start AFTER mysql?

Update:

Since I got already a few comments, here a clarification:

Apache is started as a sysvinit script under /etc/rc*.d/ whereas mysql is an upstart script under /etc/init/. Mysql isn't listed under /etc/rc*.d and thus I can't change the priority by changing the alphabetical order!

On my Ubuntu 10.04.4 server apache2 2.2.14 is installed. A few months ago starting apache2 at boot worked fine.

Now I've found out that after a reboot apache2 doesn't start automatically anymore.

The init.d script is present and should be the one from the default installation:

root@ser:~# ls /etc/init.d | grep apache
apache2

the runlevel startups are also set (using update-rc.d apache2 defaults):

root@ser:~# find /etc/rc* -name *apache*
/etc/rc0.d/K20apache2
/etc/rc1.d/K20apache2
/etc/rc2.d/S20apache2
/etc/rc3.d/S20apache2
/etc/rc4.d/S20apache2
/etc/rc5.d/S20apache2
/etc/rc6.d/K20apache2

Checking the status after reboot results in:

root@ser:~# service apache2 status
Apache is NOT running.

ps aux | grep apache is also empty.

Looking into /var/log/apache/error.log there are no entries at boot time. cat /var/log/syslog | grep apache is also empty and syslog doesn't contain any suspiscious entries.

Starting apache after boot manually with service apache2 start works fine and doesn't output any errors.

Update 1: The /etc/init.d/apache2 script has the following header/requirements:

### BEGIN INIT INFO
# Provides:          apache2
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# X-Interactive:     true
# Short-Description: Start/stop apache2 web server
### END INIT INFO

And the services being started are:

root@ser:~# ls /etc/rc2.d/
README           S10sysklogd  S20apache2 S20denyhosts  S20hashcash-milter  S20ido2db     S20modules_dep.sh  S20postfix  S20saslauthd  S20vzquota  S50rsync     S99rc.local
S09hostname_vps  S15bind9     S20exim4      S20icinga           S20memcached  S20opendkim        S20psad     S20xinetd   S23ntp      S99ondemand

How can I find out why apache2 isn't getting started at boot?

On my Ubuntu 10.04.4 server, apache2 2.2.14 eats up all the memory until there is no more left because it isn't shutting down old instances.

The server serves a small webpage which gets about one request each 3 seconds or so (from clients, google bot, scripts, etc).

If I look into the process list, there are a lot of apache2 instances:

ps aux | grep apache2

www-data  1324  0.0  6.4 580688 135600 ?       Sl   07:44   0:00 /usr/sbin/apache2 -k start
www-data  1407  0.0  6.2 518800 131748 ?       Sl   07:49   0:00 /usr/sbin/apache2 -k start
www-data  1464  0.0  6.0 518800 127740 ?       S    07:52   0:00 /usr/sbin/apache2 -k start
www-data  1492  0.0  6.1 580688 129852 ?       S    07:56   0:00 /usr/sbin/apache2 -k start
www-data  2043  0.0  6.2 501384 130864 ?       Sl   08:46   0:00 /usr/sbin/apache2 -k start
www-data  3207  0.0  6.1 501384 128260 ?       S    08:57   0:00 /usr/sbin/apache2 -k start
www-data  5711  0.0  6.2 509580 130888 ?       Sl   10:48   0:00 /usr/sbin/apache2 -k start
www-data  5779  0.0  6.0 509580 126120 ?       S    10:52   0:00 /usr/sbin/apache2 -k start
www-data  5796  0.0  6.1 491140 128660 ?       Sl   10:54   0:00 /usr/sbin/apache2 -k start
www-data  5878  0.0  6.2 510604 130144 ?       Sl   10:59   0:00 /usr/sbin/apache2 -k start
www-data  7854  0.0  6.1 492556 129712 ?       Sl   11:55   0:00 /usr/sbin/apache2 -k start
www-data  7925  0.0  5.9 492556 125592 ?       S    12:00   0:00 /usr/sbin/apache2 -k start
www-data  8018  0.0  6.3 519204 132132 ?       Sl   12:07   0:00 /usr/sbin/apache2 -k start
www-data  8019  0.0  6.5 512448 136388 ?       Sl   12:07   0:00 /usr/sbin/apache2 -k start
www-data  8020  0.0  5.8 484340 122532 ?       S    12:07   0:00 /usr/sbin/apache2 -k start
root      8159  0.0  0.0   7644   908 ttyp0    R+   12:16   0:00 grep --color=auto apache2
root     23949  0.0  6.3 483316 132176 ?       Ss   00:16   0:15 /usr/sbin/apache2 -k start
root     23951  0.0  0.0  16956   512 ?        Ss   00:16   0:00 /usr/sbin/htcacheclean -n -d120 -i -p/var/cache/apache2/mod_disk_cache -l300M
www-data 24064  0.0  6.6 590932 138868 ?       Sl   00:20   0:01 /usr/sbin/apache2 -k start
www-data 24065  0.0  6.5 533592 137812 ?       Sl   00:20   0:00 /usr/sbin/apache2 -k start
www-data 24068  0.0  6.3 500408 132372 ?       Sl   00:20   0:00 /usr/sbin/apache2 -k start
www-data 24357  0.0  6.5 525444 138352 ?       Sl   00:47   0:00 /usr/sbin/apache2 -k start
www-data 24389  0.0  6.3 533592 133604 ?       S    00:50   0:00 /usr/sbin/apache2 -k start
www-data 24527  0.0  6.5 525396 137028 ?       Sl   01:06   0:00 /usr/sbin/apache2 -k start
www-data 26006  0.0  6.3 525396 132996 ?       S    01:48   0:00 /usr/sbin/apache2 -k start
www-data 26023  0.0  6.3 525396 133952 ?       S    01:50   0:00 /usr/sbin/apache2 -k start
www-data 26304  0.0  6.6 528468 140504 ?       Sl   02:18   0:00 /usr/sbin/apache2 -k start
www-data 26305  0.0  6.4 523348 135396 ?       Sl   02:18   0:00 /usr/sbin/apache2 -k start
www-data 27902  0.0  6.4 590932 134240 ?       S    02:49   0:00 /usr/sbin/apache2 -k start
www-data 27912  0.0  6.2 523348 130988 ?       S    02:51   0:00 /usr/sbin/apache2 -k start
www-data 27942  0.0  6.4 523348 134744 ?       Sl   02:54   0:00 /usr/sbin/apache2 -k start
www-data 28508  0.0  6.3 523348 132292 ?       S    03:51   0:00 /usr/sbin/apache2 -k start
www-data 28518  0.0  6.4 528468 134736 ?       S    03:52   0:00 /usr/sbin/apache2 -k start
www-data 29919  0.0  6.3 518800 132240 ?       Sl   04:28   0:00 /usr/sbin/apache2 -k start
www-data 30164  0.0  6.1 518800 128652 ?       S    04:52   0:00 /usr/sbin/apache2 -k start
www-data 30168  0.0  6.1 518800 129576 ?       S    04:53   0:00 /usr/sbin/apache2 -k start
www-data 30418  0.0  6.7 528468 140668 ?       Sl   05:17   0:00 /usr/sbin/apache2 -k start
www-data 30463  0.0  6.3 518800 132128 ?       Sl   05:21   0:00 /usr/sbin/apache2 -k start
www-data 31799  0.0  6.1 518800 129268 ?       S    05:52   0:00 /usr/sbin/apache2 -k start
www-data 31813  0.0  6.4 528468 134808 ?       S    05:54   0:00 /usr/sbin/apache2 -k start
www-data 31881  0.0  6.4 515152 135252 ?       Sl   06:00   0:00 /usr/sbin/apache2 -k start
www-data 32005  0.0  6.2 517776 131220 ?       Sl   06:12   0:00 /usr/sbin/apache2 -k start
www-data 32405  0.0  6.0 517776 127832 ?       S    06:52   0:00 /usr/sbin/apache2 -k start
www-data 32435  0.0  6.1 515152 129456 ?       S    06:55   0:00 /usr/sbin/apache2 -k start

Apache server status (ExtendedStatus OFF) gives:

Server Version: Apache/2.2.14 (Ubuntu) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_perl/2.0.4 Perl/v5.10.1
Server Built: Mar 8 2013 16:46:35

Current Time: Wednesday, 05-Jun-2013 12:17:22 CEST
Restart Time: Wednesday, 05-Jun-2013 00:16:19 CEST
Parent Server Generation: 2
Server uptime: 12 hours 1 minute 3 seconds
19 requests currently being processed, 2 idle workers

GGGGGGGGGGGGGG.GWG_G_..W........................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

PID Key:

   24527 in state: G ,   27942 in state: G ,   5796 in state: G 
   26304 in state: G ,   26305 in state: G ,   30418 in state: G 
   24064 in state: G ,   24065 in state: G ,   31881 in state: G 
   32005 in state: G ,   24068 in state: G ,   29919 in state: G 
   1324 in state: G ,   30463 in state: G ,   5711 in state: G 
   8018 in state: W ,   1407 in state: G ,   8019 in state: _ 
   2043 in state: G ,   8020 in state: _ ,   7854 in state: W 

To obtain a full report with current status information you need to use the ExtendedStatus On directive.
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 0
subcaches: 32, indexes per subcache: 133
index usage: 0%, cache usage: 0%
total sessions stored since starting: 4
total sessions expired since starting: 4
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 0 hit, 0 miss
total removes since starting: 0 hit, 0 miss

So you can see that most of the processes hang in the G status (Gracefully finishing).

Update 1: Using apache server status with "ExtendedStatus On" we see that the processes are stuck in the W status (Sending reply): (I replaced the true IPs with ipX.domain.com)

Server Version: Apache/2.2.14 (Ubuntu) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_perl/2.0.4 Perl/v5.10.1
Server Built: Mar 8 2013 16:46:35

Current Time: Wednesday, 05-Jun-2013 18:49:48 CEST
Restart Time: Wednesday, 05-Jun-2013 14:31:17 CEST
Parent Server Generation: 0
Server uptime: 4 hours 18 minutes 31 seconds
Total accesses: 3129 - Total Traffic: 111.9 MB
CPU Usage: u2.29 s.18 cu0 cs0 - .0159% CPU load
.202 requests/sec - 7.4 kB/second - 36.6 kB/request
6 requests currently being processed, 2 idle workers

WWW.W._WW..._...................................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv     PID     Acc         M   CPU     SS      Req     Conn    Child   Slot    Client          VHost           Request
0-0     7926    1/28/390    W   0.00    2913    0       0.2     1.15    11.63   ip1.domain.com  www.domain.com  GET /de HTTP/1.1
1-0     5432    0/10/183    W   0.00    10005   0       0.0     0.59    7.51    ip6.domain.com  def.domain.com  HEAD / HTTP/1.1
2-0     7878    1/44/403    W   0.00    2710    0       0.2     2.29    13.22   ip1.domain.com  www.domain.com  GET /de HTTP/1.1
3-0     -       0/0/467     .   0.00    203     0       0.0     0.00    19.51   ip2.domain.com  def.domain.com  NULL
4-0     5854    1/58/223    W   0.01    6137    0       0.2     1.74    12.29   ip1.domain.com  www.domain.com  GET /de HTTP/1.1
5-0     -       0/0/359     .   1.40    204     0       0.0     0.00    10.42   ip2.domain.com  def.domain.com  NULL
6-0     9534    0/60/244    _   0.63    69      0       0.0     1.60    6.70    ip1.domain.com  def.domain.com  NULL
7-0     5861    1/32/84     W   0.00    6746    0       0.2     1.55    3.18    ip1.domain.com  www.domain.com  GET /de HTTP/1.1
8-0     11401   98/106/440  W   0.17    0       0       189.2   0.38    15.68   ip5.domain.com  def.domain.com  GET /server-status HTTP/1.1
9-0     -       0/0/152     .   0.00    207     1608    0.0     0.00    8.06    ip2.domain.com  def.domain.com  NULL
10-0    -       0/0/31      .   0.00    206     1639    0.0     0.00    0.79    ip2.domain.com  def.domain.com  NULL
11-0    -       0/0/99      .   0.24    205     0       0.0     0.00    1.83    ip2.domain.com  def.domain.com  NULL
12-0    11404   0/4/24      _   0.02    104     18      0.0     0.00    0.38    ip3.domain.com  www.domain.com  GET /robots.txt HTTP/1.1
13-0    -       0/0/8       .   0.00    208     0       0.0     0.00    0.06    ip1.domain.com  www.domain.com  GET /de HTTP/1.1
14-0    -       0/0/16      .   0.00    920     47      0.0     0.00    0.59    ip4.domain.com  def.domain.com  NULL
15-0    -       0/0/2       .   0.00    5661    3905    0.0     0.00    0.08    ip2.domain.com  def.domain.com  NULL
16-0    -       0/0/3       .   0.00    5686    0       0.0     0.00    0.00    ip2.domain.com  def.domain.com  NULL
17-0    -       0/0/1       .   0.00    5694    1       0.0     0.00    0.00    127.0.0.1       def.domain.com  GET / HTTP/1.0
Srv Child Server number - generation
PID OS process ID
Acc Number of accesses this connection / this child / this slot
M   Mode of operation
CPU CPU usage, number of seconds
SS  Seconds since beginning of most recent request
Req Milliseconds required to process most recent request
Conn    Kilobytes transferred this connection
Child   Megabytes transferred this child
Slot    Total megabytes transferred this slot
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 0
subcaches: 32, indexes per subcache: 133
index usage: 0%, cache usage: 0%
total sessions stored since starting: 37
total sessions expired since starting: 37
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 8 hit, 0 miss
total removes since starting: 0 hit, 0 miss

As you can see here, there are a few processes with a very high SS (Time since last request) value which means they are stuck. The requests were for different VHosts and also from different clients.

If I look into the acces.log files and search for the entries for such a process (I added the PID to log), the last request they served was most of the times a simple image file directly elaborated by apache.

The important parts of the apache2 config are:

<IfModule mpm_prefork_module>
    StartServers       1
    MinSpareServers    1
    MaxSpareServers    3
    MaxClients         30
    MaxRequestsPerChild   0
</IfModule>
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

Update 3: All the enabled apache2 modules:

root@ser:~# apache2ctl -M
Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cache_module (shared)
 cgi_module (shared)
 deflate_module (shared)
 dir_module (shared)
 disk_cache_module (shared)
 env_module (shared)
 expires_module (shared)
 filter_module (shared)
 headers_module (shared)
 mem_cache_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 version_module (shared)
 pagespeed_module (shared)
 perl_module (shared)
 php5_module (shared)
 proxy_module (shared)
 proxy_http_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 ssl_module (shared)
 status_module (shared)
 unique_id_module (shared)
Syntax OK

Update 2: Here is the stacktrace of a hanging process. All of the stacktraces of the stuck ones are exactly the same:

5861:/usr/sbin/apache2-kstart
#0  0x00002ad420a4448d in /lib/libpthread-2.11.1.so: waitpid
#1  0x00002ad4258dc507 in /usr/lib/libperl.so.5.10.1: Perl_wait4pid
#2  0x00002ad4258dc760 in /usr/lib/libperl.so.5.10.1: Perl_my_pclose
#3  0x00002ad425949b1d in /usr/lib/libperl.so.5.10.1: Perl_io_close
#4  0x00002ad425949c89 in /usr/lib/libperl.so.5.10.1: Perl_do_close
#5  0x00002ad42593d922 in /usr/lib/libperl.so.5.10.1: Perl_pp_close
#6  0x00002ad4258f3346 in /usr/lib/libperl.so.5.10.1: Perl_runops_standard
#7  0x00002ad42589a8cf in /usr/lib/libperl.so.5.10.1: Perl_call_sv
#8  0x00002ad4256202f6 in /usr/lib/apache2/modules/mod_perl.so: modperl_callback
#9  0x00002ad425620a3a in /usr/lib/apache2/modules/mod_perl.so: modperl_callback_run_handlers
#10 0x00002ad42562101f in /usr/lib/apache2/modules/mod_perl.so: modperl_callback_per_dir
#11 0x00002ad42561a99f in /usr/lib/apache2/modules/mod_perl.so: -
#12 0x00002ad42561ad06 in /usr/lib/apache2/modules/mod_perl.so: modperl_response_handler
#13 0x00002ad41ff58320 in /usr/lib/apache2/mpm-prefork/apache2: ap_run_handler (/build/buildd/apache2-2.2.14/server/config.c:159)
#14 0x00002ad41ff5bc88 in /usr/lib/apache2/mpm-prefork/apache2: ap_invoke_handler (/build/buildd/apache2-2.2.14/server/config.c:377)
#15 0x00002ad41ff69878 in /usr/lib/apache2/mpm-prefork/apache2: ap_process_request (/build/buildd/apache2-2.2.14/modules/http/http_request.c:282)
#16 0x00002ad41ff66728 in /usr/lib/apache2/mpm-prefork/apache2: ap_process_http_connection (/build/buildd/apache2-2.2.14/modules/http/http_core.c:190)
#17 0x00002ad41ff5fed8 in /usr/lib/apache2/mpm-prefork/apache2: ap_run_process_connection (/build/buildd/apache2-2.2.14/server/connection.c:43)
#18 0x00002ad41ff6e847 in /usr/lib/apache2/mpm-prefork/apache2: child_main (/build/buildd/apache2-2.2.14/server/mpm/prefork/prefork.c:663)
#19 0x00002ad41ff6eb5a in /usr/lib/apache2/mpm-prefork/apache2: make_child (/build/buildd/apache2-2.2.14/server/mpm/prefork/prefork.c:758)
#20 0x00002ad41ff6f7e4 in /usr/lib/apache2/mpm-prefork/apache2: perform_idle_server_maintenance (/build/buildd/apache2-2.2.14/server/mpm/prefork/prefork.c:883)
#21 0x00002ad41ff44350 in /usr/lib/apache2/mpm-prefork/apache2: main (/build/buildd/apache2-2.2.14/server/main.c:742)
#22 0x00002ad420c70c4d in /lib/libc-2.11.1.so: __libc_start_main@@GLIBC_2.2.5
#23 0x00002ad41ff432e9 in /usr/lib/apache2/mpm-prefork/apache2: _start

For comaprison the stacktrace of a process which doesn't hang:

9741:/usr/sbin/apache2-kstart
#0  0x00002ad420d3d7d7 in /lib/libc-2.11.1.so: semop@@GLIBC_2.2.5
#1  0x00002ad42081a0c3 in /usr/lib/libapr-1.so.0.3.8: -
#2  0x00002ad42081ae47 in /usr/lib/libapr-1.so.0.3.8: apr_proc_mutex_lock
#3  0x00002ad41ff6e754 in /usr/lib/apache2/mpm-prefork/apache2: accept_mutex_on (/build/buildd/apache2-2.2.14/server/mpm/prefork/prefork.c:206)
#4  0x00002ad41ff6eb5a in /usr/lib/apache2/mpm-prefork/apache2: make_child (/build/buildd/apache2-2.2.14/server/mpm/prefork/prefork.c:758)
#5  0x00002ad41ff6f7e4 in /usr/lib/apache2/mpm-prefork/apache2: perform_idle_server_maintenance (/build/buildd/apache2-2.2.14/server/mpm/prefork/prefork.c:883)
#6  0x00002ad41ff44350 in /usr/lib/apache2/mpm-prefork/apache2: main (/build/buildd/apache2-2.2.14/server/main.c:742)
#7  0x00002ad420c70c4d in /lib/libc-2.11.1.so: __libc_start_main@@GLIBC_2.2.5
#8  0x00002ad41ff432e9 in /usr/lib/apache2/mpm-prefork/apache2: _start

Here my Question:

How can I find out what blocks the apache instances from shutting down successfully?

Update 4 Googling for 'apache sending reply hung' results in a few posts where people had problems with buggy scripts/mysql connections. As it can be seen in the stacktrace, the processes which got stuck did use mod_perl, maybe there's a problem... Will dig into it

I'm running logwatch and there I've seen some strange activity.

There are two different ip adresses (I'll call this guy hacker) where it seems they logged in successfully through vsdftpd: hacker's IP 1: 91.121.106.53

vsftpd.log.1:Tue Feb 19 20:46:18 2013 [pid 24450] CONNECT: Client "91.121.106.53"
vsftpd.log.1:Tue Feb 19 20:46:18 2013 [pid 24449] [public] OK LOGIN: Client "91.121.106.53"
vsftpd.log.1:Tue Feb 19 20:46:18 2013 [pid 24451] [public] OK UPLOAD: Client "91.121.106.53", "//1c.php", 23 bytes, 0.51Kbyte/sec
vsftpd.log.1:Tue Feb 19 20:46:18 2013 [pid 24451] [public] OK DELETE: Client "91.121.106.53", "//1c.php"
vsftpd.log.1:Wed Feb 20 03:21:59 2013 [pid 3610] CONNECT: Client "91.121.106.53"
vsftpd.log.1:Wed Feb 20 03:21:59 2013 [pid 3609] [webcam] OK LOGIN: Client "91.121.106.53"
vsftpd.log.1:Wed Feb 20 03:40:20 2013 [pid 3897] CONNECT: Client "91.121.106.53"
vsftpd.log.1:Wed Feb 20 03:40:20 2013 [pid 3896] [public] OK LOGIN: Client "91.121.106.53"

And hacker's IP 2: 72.52.172.4

vsftpd.log.1:Sun Feb 17 09:36:00 2013 [pid 20290] CONNECT: Client "72.52.172.4"
vsftpd.log.1:Sun Feb 17 09:36:02 2013 [pid 20289] [MyUserName] FAIL LOGIN: Client "72.52.172.4"
vsftpd.log.1:Sun Feb 17 09:45:23 2013 [pid 20404] CONNECT: Client "72.52.172.4"
vsftpd.log.1:Sun Feb 17 09:45:23 2013 [pid 20403] [webcam] OK LOGIN: Client "72.52.172.4"
vsftpd.log.1:Sun Feb 17 10:03:04 2013 [pid 21690] CONNECT: Client "72.52.172.4"
vsftpd.log.1:Sun Feb 17 10:03:05 2013 [pid 21689] [public] OK LOGIN: Client "72.52.172.4"
vsftpd.log.1:Mon Feb 18 02:09:29 2013 [pid 17459] CONNECT: Client "72.52.172.4"
vsftpd.log.1:Mon Feb 18 02:09:29 2013 [pid 17458] [public] OK LOGIN: Client "72.52.172.4"
vsftpd.log.1:Mon Feb 18 03:09:40 2013 [pid 18426] CONNECT: Client "72.52.172.4"
vsftpd.log.1:Mon Feb 18 03:09:41 2013 [pid 18425] [webcam] OK LOGIN: Client "72.52.172.4"

The only user which currently uses FTP is the webcam user which is a script.

The next strange thing is, that the hacker knew exactly, which users exist. As you can see here, the other hackers normally try 'anonymous' and that's it:

vsftpd.log:Sun Feb 24 11:16:17 2013 [pid 26192] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log:Sun Feb 24 11:16:20 2013 [pid 26194] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Sun Feb 17 08:34:52 2013 [pid 18379] [anonymous] FAIL LOGIN: Client "66.249.76.54"
vsftpd.log.1:Sun Feb 17 08:34:55 2013 [pid 18381] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Sun Feb 17 09:36:02 2013 [pid 20289] [MyUserName] FAIL LOGIN: Client "72.52.172.4"
vsftpd.log.1:Mon Feb 18 14:41:28 2013 [pid 5601] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Mon Feb 18 14:41:31 2013 [pid 5604] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Mon Feb 18 16:01:54 2013 [pid 7801] [anonymous] FAIL LOGIN: Client "69.162.83.5"
vsftpd.log.1:Tue Feb 19 16:13:10 2013 [pid 16375] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Tue Feb 19 16:13:13 2013 [pid 16377] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Wed Feb 20 15:08:44 2013 [pid 23754] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Wed Feb 20 15:08:46 2013 [pid 23756] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Thu Feb 21 18:09:50 2013 [pid 5589] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.1:Thu Feb 21 18:09:52 2013 [pid 5591] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.2:Sun Feb 10 11:41:42 2013 [pid 26587] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.2:Sun Feb 10 11:41:45 2013 [pid 26589] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.2:Mon Feb 11 06:15:10 2013 [pid 26421] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.2:Thu Feb 14 17:05:30 2013 [pid 8180] [anonymous] FAIL LOGIN: Client "66.249.76.54"
vsftpd.log.2:Thu Feb 14 17:05:34 2013 [pid 8182] [anonymous] FAIL LOGIN: Client "66.249.76.54"
vsftpd.log.2:Fri Feb 15 02:36:18 2013 [pid 24301] [anonymous] FAIL LOGIN: Client "66.249.76.54"
vsftpd.log.2:Fri Feb 15 18:36:11 2013 [pid 19947] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.2:Fri Feb 15 18:36:14 2013 [pid 19954] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.2:Sat Feb 16 08:07:01 2013 [pid 9810] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.3:Sun Feb  3 17:27:32 2013 [pid 7448] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.3:Sun Feb  3 17:27:36 2013 [pid 7450] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.3:Mon Feb  4 11:28:34 2013 [pid 5688] [anonymous] FAIL LOGIN: Client "66.249.75.54"
vsftpd.log.3:Tue Feb  5 04:53:02 2013 [pid 3520] [anonymous] FAIL LOGIN: Client "66.249.76.54"
vsftpd.log.3:Tue Feb  5 04:53:05 2013 [pid 3522] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.3:Wed Feb  6 23:08:28 2013 [pid 13439] [anonymous] FAIL LOGIN: Client "66.249.75.54"
vsftpd.log.3:Wed Feb  6 23:08:32 2013 [pid 13441] [anonymous] FAIL LOGIN: Client "66.249.75.54"
vsftpd.log.3:Thu Feb  7 18:09:10 2013 [pid 13644] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.3:Thu Feb  7 18:09:14 2013 [pid 13646] [anonymous] FAIL LOGIN: Client "66.249.75.54"
vsftpd.log.3:Fri Feb  8 12:22:41 2013 [pid 11563] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.3:Fri Feb  8 12:22:44 2013 [pid 11565] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.3:Sat Feb  9 06:37:37 2013 [pid 9738] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.3:Sat Feb  9 06:37:40 2013 [pid 9744] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.4:Mon Jan 28 08:50:29 2013 [pid 30389] [anonymous] FAIL LOGIN: Client "66.249.76.54"
vsftpd.log.4:Mon Jan 28 08:50:32 2013 [pid 30392] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.4:Tue Jan 29 12:56:26 2013 [pid 15682] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.4:Tue Jan 29 12:56:29 2013 [pid 15684] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.4:Wed Jan 30 10:43:11 2013 [pid 21831] [anonymous] FAIL LOGIN: Client "66.249.75.54"
vsftpd.log.4:Wed Jan 30 10:43:15 2013 [pid 21833] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.4:Fri Feb  1 00:07:50 2013 [pid 22202] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.4:Fri Feb  1 00:07:53 2013 [pid 22204] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.4:Fri Feb  1 11:40:15 2013 [pid 9412] [anonymous] FAIL LOGIN: Client "66.249.76.54"
vsftpd.log.4:Sat Feb  2 15:03:42 2013 [pid 25912] [anonymous] FAIL LOGIN: Client "66.249.78.54"
vsftpd.log.4:Sat Feb  2 15:03:46 2013 [pid 25914] [anonymous] FAIL LOGIN: Client "66.249.78.54"

So how is it possible that the the hacker knew exactly the only three existing users (public, webcam, MyUserName) which exist on vsftpd and was able to log in with public and webcam?

My server is a virtual hosted server in Germany with Ubuntu Server Ubuntu 10.04.4 LTS with vsftpd version 2.2.2 (yes, I know that this version is old, I'll update it immediately)

The only possibilities I can think of is:

• Package sniffing: The hacker sniffed the connection of the server, but I've used the public user only once two years ago thus there Is no possibility that the hacker knew this user name. (SFTP wasn't forced, but it've forced it from now on)
• Brute force: The default options max_login_fails=3 and delay_failed_login=1 should prevent this. I've now added them explicitly to the configuration of vsftpd.
• Backdoor in vsftpd: how can I analyze this?

I'll now restrict the access to the IP address of the script (because only this needs FTP, there are no other users).

Would be nice if you can give me some tips on how I can analyze this problem (how was it possible for the hacker to login).

Thanks!

I configured my postfix server to relay all received mails to smtp.example.com with relayhost. This server doesn't accept mails from an external source. So only scripts are sending mails.

Is it possible to configure postfix that all mails with a specific mail address in FROM field, like [email protected] (or also the whole domain: domain.net) are send directly by the server to the corresponding mx server for the TO-address, instead of using the specified relay host?

I found one possibility to specify a specific host if a specific mail address is in the TO field, but not the FROM field: How can I configure Postfix to ignore relayhost for some domains? and http://www.postfix.org/transport.5.html

Thanks!!

How can I remove the Administrator Mailbox from Exchange 2010? I don't want to delete the whole user account because I need Administrator to access the Server, but Administrator shouldn't have any mail addresses assigned.

Background: We use Eset Mail Security which is getting licenced per mailbox count. There is also a tool from ESET to check current mailbox count: http://kb.eset.com/esetkb/index?page=content&id=SOLN2425 To avoid paying for unused mailboxes, I want to disable Administrator's (and also other unused) mail boxes.

Thanks!