Today I noticed unusual high request rate on Apache webserver and also quite high incoming network traffic. Upon checking Apache's mod_status page, I found the offending URLs to be from path www.server.com/www/wp-includes/js/tinymce/plugins/wpautoresize/
. And indeed I have found several hacked (obfuscated) PHP scripts there.
Also noticed weird process executed by www-data user:
www-data 7300 10.8 0.1 2122900 18768 ? Ssl Jul11 121:47 /usr/bin/host
Checking /proc/7300/cmdline
revealed that indeed this is the original /usr/bin/host
binary. netstat -anp
showed it has many HTTP connections opened, so somehow that binary is abused. debsums
confirmed the binary checksum to be OK. As the process was run under www-data user, I had no reason to believe server itself was compromised.
How is that binary abused?
EDIT: This not broad "how to deal with compromised server" question. Rather a question (and already an answer) about one specific type of abuse how is it technically done, as this particular case is quite creative in how it works. It seems this is in wild for several years now (old threads & questions from 2012) and I encountered it this week.
After digging in the source codes of offended PHP scripts and googling (this thread), I have found an explanation.
This is part of
system.php
code which I have found:How
/usr/bin/host
is involved is a bit more advanced. Programs use libraries (.so
files) for some of their functions. Users can prelink (LD_PRELOAD
) some .so file(s) before launching a legitimate binary to change how it acts.As you can see this script creates a file
libworker.so
and usesLD_PRELOAD
environment variable to preload it, so legitimatehost
binary is doing something totally different.It creates a
1.sh
shell script and tries to execute it in several ways (directly, usingat
command, using cron). Immediately after that it removes the script and library file from disk, so it gets un-noticed.What happened in the first place was that some vulnerable Wordpress plugin got abused and attacker was able to put their files into word-writabble directories.
Mitigation means analyzing old access log files for that domain and trying to find any
POST
requests to unusual locations - for example directly accessing WP/Joomla plugin's PHP files is unusual. Then remove all found obfuscated PHP files, fix directory permissions, terminate runninghost
processes and monitor logfiles for any attemps of re-hacks.EDIT: I've got info from ESET that they alredy detect this particular library and also other versions. Antivirus companies give it name Roopre and it seems that it is used as part of Mayhem botnet.
In-depth analysis of the Mayhem botnet.
In-depth analysis of this exploit.