Home / server / Questions / 359440
Accepted
tomlogic
Asked: 2012-02-13 19:56:11 +0800 CST

Finding source of directory traversal attack originating from my server

  • 772

Two weeks ago, I was notified by my VPS provider that my server (CentOS 5.5, yum is up to date) had originated "NULL byte/Directory Traversal" attacks agains some servers at DreamHost. I spent a few hours going over the server with a fine toothed comb and didn't find anything. Before logging in, I retrieved a copy of the sshd binary and confirmed that it hadn't been modified. I installed a rootkit checker (chkrootkit-0.49) that found nothing. I checked the web logs of the websites I host, looking for a hit that may have triggered a script on my server to initiate the attacks but found nothing. Checked /var/log/secure and /var/log/messages around the times of the attack but found nothing. Checked last, but found nothing. Did a find on key directories looking for files modified in the past 3 days, but nothing.

What else can I do to to find the cause of the attacks? I wrote a script to check for outbound TCP connections on port 80, but only came up with legitimate connections (SpamAssassin and ClamAV downloading updates, Joomla checking its site for updates, etc.). Even if I did see an active outbound connection, would I even be able to dump data from the process (in the /proc directory) to show me the account originating the attack?

After watching the server for a few days, I gave up. Now I've received another complaint from DreamHost, so it's happened again. I've requested detailed logs from DreamHost, but then what? Where else can I look? If I can't find the source, is there something I can install to monitor the server and log data when it starts making outbound connections to tcp/80 in the DreamHost IP space? What would I log? Just get a tcpdump of all traffic in that timeframe and try to sift through it manually?

Update

See my accepted answer for the solution I came up with.

I'm still interested in options for logging the source of all outbound port 80 traffic -- a way to know what the source process is and perhaps its parent process (and the parent's parent).

linux centos security

2 Answers

  1. Well, first thing you should have done if not already done, is change your root password (and any other shell-enabled accounts). Second, I would probably install DenyHosts, without synchronizing it with the live list, and watch /etc/hosts.deny for a while. Make sure to add your own IP address to DenyHosts's allowed_hosts file in case you don't have console access to unlock yourself if you mistype your password more than 3 times... Also, set sshd_config to disallow direct root login if not already done, and use sudo.

    If you're able to, you could also redirect syslog to an external server to prevent the hacker from deleting his traces manually from the logs. You could also write a simple script that could be called from .bashrc of every user so that it sends you an email everytime someone logs in, and appends useful info such as ip adresses (detailed who command) and maybe even attach the last few lines of important logs. That way, a hacker would have to be very quick to prevent that from firing.

    That alone should make sure "nobody" but you can actively login to your system with a shell at least (well... it will block 99% of attackers anyway).

    Then you could always look into file integrity/intrusion detection systems such as Samhain to really see what's going on.

    Here's another useful link to monitor your system: 20 Linux System Monitoring Tools Every SysAdmin Should Know

    Good luck and I hope this helped!

    • 0
  2. Best Answer

    Even though VirtualHosts log to their own error_log files, I found output from wget in /var/log/httpd/error_log. I ended up wrapping wget with the following shell script:

    #!/bin/sh

MYPID=$$
MYPWD=`pwd`
PARENT=`ps -o ppid -p $$ | tail -1 | bc`
GP=`ps -o ppid -p ${PARENT} | tail -1 | bc`
WHOAMI=`whoami`

TMPFILE=/tmp/dump.$$

function cat_file() {
    local fn="$1"

    echo "dump of $fn :" >> ${TMPFILE}
    cat $fn >> ${TMPFILE}
}

function proc_dump() {
    local pdir="/proc/$1"

    echo "Dumping info on process $1" >> ${TMPFILE}
    ls -alR ${pdir} >> ${TMPFILE}
    cat_file "${pdir}/cmdline"
    echo "environment" >> ${TMPFILE}
    xargs --null --max-args=1 echo < ${pdir}/environ >> ${TMPFILE}
    cat_file "${pdir}/loginuid"
    cat_file "${pdir}/maps"
}

echo "I am ${WHOAMI}, running from ${MYPWD}, pid ${MYPID}, ppid ${PARENT}" > ${TMPFILE}
echo "arguments: $@" >> ${TMPFILE}
export >> ${TMPFILE}
proc_dump ${MYPID}
proc_dump ${PARENT}
proc_dump ${GP}

mail -s "wget call on foo" [email protected] < ${TMPFILE}

exec /usr/bin/wget.real "$@"

    From that, I was able to find out which VirtualHost was responsible, look closely at its logs, and figure out that there was an old osCommerce directory in the site's directory tree, which I ultimately removed.

    I've left the wrapped wget in there in case it can help me with any future attacks.

    • 0