tomlogic's questions

CentOS 6.2, bind 9.7.3, rsyslog 4.6.2

I recently set up a server, and I noticed that named had stopped logging to /var/log/messages after the logs had rotated. I thought that was odd, since all logging happens through rsyslog and named doesn't write directly to the log file.

It was even more odd because I had HUPed named after updating a zone file, and it still wasn't logging.

After I stopped and restarted named, logging resumed.

What's going on here? The syslog PID hasn't changed (/var/run/syslogd.pid matches the PID shown in ps). Is rsyslog opening a new socket when logrotate rotates its logs and HUPs it?

/etc/logrotate.d/syslog:

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

And rsyslogd continues logging after the rotate -- it's just named (and possibly other services) that aren't connecting to rsyslogd correctly. Here's /var/log/messages-20120212:

Feb  5 04:45:03 linus kernel: imklog 4.6.2, log source = /proc/kmsg started.
Feb  5 04:45:03 linus rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1178" x-info="http://www.rsyslog.com"] (re)start
Feb  9 16:16:29 linus yum[13028]: Installed: 2:vim-minimal-7.2.411-1.6.el6.i686
Feb 12 04:09:22 linus rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1178" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'restart'.
Feb 12 04:09:22 linus kernel: Kernel logging (proc) stopped.

And I'm only running rsyslogd (which for some reason saves its PID in /var/run/syslogd.pid):

root      1178  0.0  0.1  29496  1488 ?        Sl   Jan29   0:20 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
root      5960  0.0  0.0   2300   504 pts/0    S+   17:45   0:00 grep syslog

Two weeks ago, I was notified by my VPS provider that my server (CentOS 5.5, yum is up to date) had originated "NULL byte/Directory Traversal" attacks agains some servers at DreamHost. I spent a few hours going over the server with a fine toothed comb and didn't find anything. Before logging in, I retrieved a copy of the sshd binary and confirmed that it hadn't been modified. I installed a rootkit checker (chkrootkit-0.49) that found nothing. I checked the web logs of the websites I host, looking for a hit that may have triggered a script on my server to initiate the attacks but found nothing. Checked /var/log/secure and /var/log/messages around the times of the attack but found nothing. Checked last, but found nothing. Did a find on key directories looking for files modified in the past 3 days, but nothing.

What else can I do to to find the cause of the attacks? I wrote a script to check for outbound TCP connections on port 80, but only came up with legitimate connections (SpamAssassin and ClamAV downloading updates, Joomla checking its site for updates, etc.). Even if I did see an active outbound connection, would I even be able to dump data from the process (in the /proc directory) to show me the account originating the attack?

After watching the server for a few days, I gave up. Now I've received another complaint from DreamHost, so it's happened again. I've requested detailed logs from DreamHost, but then what? Where else can I look? If I can't find the source, is there something I can install to monitor the server and log data when it starts making outbound connections to tcp/80 in the DreamHost IP space? What would I log? Just get a tcpdump of all traffic in that timeframe and try to sift through it manually?

Update

See my accepted answer for the solution I came up with.

I'm still interested in options for logging the source of all outbound port 80 traffic -- a way to know what the source process is and perhaps its parent process (and the parent's parent).

I've seen an increase in spam recently, and a lot of it has included .html attachments. Some emails masquerade as bounce messages, encouraging users to open the HTML to see what message bounced. Others use similar techniques to trick the user into opening the file. I'm considering blocking all email with a .html file attached, in an attempt to catch more of this spam. Using zen.spamhaus.org had greatly reduced the amount of spam received when I started using it last September, but a lot of these .html attachments have been getting through.

I took a look inside a few and they used META REFRESH tags to redirect to some website. Since the attachment is base64 encoded, SpamAssassin can't check the URL against the various URL blacklists.

I'm using SpamAssassin, ClamAV and simscan (an add-on for qmail) on my system, and can easily block certain attachments. My question is whether .html files are commonly sent as attachments on legitimate mail.

NOTE 1: I'm not referring to HTML formatted email, I'm specifically talking about a .html file as an attachment that must be opened in a web browser.

NOTE 2: I'm considering this for the hosting servers I maintain for 150+ domains. Most are small businesses with 1-5 accounts.