I just noticed a weird directories stored on both in /tmp within two of my CentOS boxes.
On one machine the temp directory is called /tmp/www4-679109 while on my second machine the temp directory is /tmp/sos_e6X9_3.
Both of these suspicious directories contain four subdirectories: etc proc sys var .
The etc folder on this suspicious directory tree contains an identical copy of my master submit.cf sendmail configuration file. /tmp/sos_e6X9_3/etc/mail/submit.cf and /etc/mail/submit.cf. This makes me think that some how sendmail was used to relay mail. Though I can't verify the maillog files were inconclusive is this was the case.
Also this suspecious wierd directory contains some sort of audit log files within it. (/tmp/sos_e6X9_3/var/log/audit/audit.log.1)
Snippet of contents of a log file:
type=LOGIN msg=audit(1293719401.416:2772543): login pid=6662 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=557197
type=LOGIN msg=audit(1293719401.417:2772544): login pid=6660 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=557198
type=LOGIN msg=audit(1293719401.418:2772545): login pid=6649 uid=0 old auid=4294967295 new auid=599 old ses=4294967295 new ses=557199
type=LOGIN msg=audit(1293719401.420:2772546): login pid=6665 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=557200
type=USER_ACCT msg=audit(1293719401.423:2772547): user pid=6668 uid=0 auid=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='PAM: accounting acct="root" : exe="/usr/sbin/crond" (hostname=?, addr=?, terminal=cron res=success)'
type=USER_START msg=audit(1293719401.424:2772548): user pid=6653 uid=0 auid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='PAM: session open acct="root" : exe="/usr/sbin/crond" (hostname=?, addr=?, terminal=cron res=success)'
What is concerning me more with this suspicious files and directories on my CentOS boxes is that it appears that their is a procfs like directory in /tmp/sos_e6X9_3/proc. The only difference with /tmp/sos_e6X9_3/proc and /proc is that in the /tmp/sos_e6X9_3/proc there aren't any viewable process information.
I have kept both CentOS machines up to date and did not make any change to sendmail that would somewhow make it vulnerable, (unless sendmail is vulnerable right off the box).
Does anyone have have any ideas/feedback on why these suspicious temp files and folders were created?
UPDATE: It appears that the suspicious files were generated by the sosreport utility when we were working with Red Hat support on a different issue. I checked the log and the timestamps match with the timestamp with the suspicious files and directories. Thanks for your support and feedback. You guys rock!
I would definitely run 'chkrootkit' and 'rkhunter' immediately. If you can download a binary for netstat, lsof, ls, less and ps and run them directly then you'll have a better idea what the boxes are doing.
Note that these downloaded, clean binaries should only be trusted for that login session. Download them each time you need them. I've seen files overwritten on each login after a compromise.
If you can firewall everything except your admin IP for SSH for a few minutes then all the better while you run these tests.
During your careful search around the machine look for ANYTHING different. From the formatted colour output of 'top' to the speed 'w' loads up.
The sos directory you found in /tmp is created when the command sosreport is executed. This is a Red Hat utility that collects system information that is useful for Red Hat Support when troubleshooting your system. This, as far as I know, isn't useful on CentOS but since CentOS is based off of RHEL that's why it's there (assumption).
Edit: I missed your bold print stating you found out it deals with sosreport.
No matter whether you actually find a rootkit, I would:
Yes, this is a bit paranoid, but is typically a faster route to revovery than trying to study the problem and be smart.
It seems from OP that this is a mail server. If so, you have an additional problem in that the box may be sending out contaminated mail. I would not release my mail queues until I was absolutely sure it was not being rewritten enroute.