So, we host a geoservice webserver thing at the office.
Someone apparently broke into this box (probably via ftp or ssh), and put some kind of irc-managed rootkit thing.
Now I'm trying to clean the whole thing up, I found the process pid who tries to connect via irc, but i can't figure out who's the invoking process (already looked with ps, pstree, lsof) The process is a perl script owned by www user, but ps aux |grep displays a fake file path on the last column.
Is there another way to trace that pid and catch the invoker?
Forgot to mention: the kernel is 2.6.23, which is exploitable to become root, but I can't touch this machine too much, so I can't upgrade the kernel
EDIT: lsof might help:
lsof -p 9481
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEss
perl 9481 www cwd DIR 8,2 608 2 /ss
perl 9481 www rtd DIR 8,2 608 2 /ss
perl 9481 www txt REG 8,2 1168928 38385 /usr/bin/perl5.8.8ss
perl 9481 www mem REG 8,2 135348 23286 /lib64/ld-2.5.soss
perl 9481 www mem REG 8,2 103711 23295 /lib64/libnsl-2.5.soss
perl 9481 www mem REG 8,2 19112 23292 /lib64/libdl-2.5.soss
perl 9481 www mem REG 8,2 586243 23293 /lib64/libm-2.5.soss
perl 9481 www mem REG 8,2 27041 23291 /lib64/libcrypt-2.5.soss
perl 9481 www mem REG 8,2 14262 23307 /lib64/libutil-2.5.soss
perl 9481 www mem REG 8,2 128642 23303 /lib64/libpthread-2.5.soss
perl 9481 www mem REG 8,2 1602809 23289 /lib64/libc-2.5.soss
perl 9481 www mem REG 8,2 19256 38662 /usr/lib64/perl5/5.8.8/x86_64-linux-threa d-multi/auto/IO/IO.soss
perl 9481 www mem REG 8,2 21328 38877 /usr/lib64/perl5/5.8.8/x86_64-linux-threa d-multi/auto/Socket/Socket.soss
perl 9481 www mem REG 8,2 52512 23298 /lib64/libnss_files-2.5.soss
perl 9481 www 0r FIFO 0,5 1068892 pipess
perl 9481 www 1w FIFO 0,5 1071920 pipess
perl 9481 www 2w FIFO 0,5 1068894 pipess
perl 9481 www 3u IPv4 130646198 TCP 192.168.90.7:60321->www.****.net:ircd (SYN_SENT)
If I can give you any advice, it is to stop wasting your time cleaning up. Make an image of the OS for forensic stuff for later, and just reinstall the server.
Sorry, but its the only secure way to resolving yourself from being rootkitted.
Later you can check the image, for certain reasons, why it happened.
From my own personal experience, I did this, and later found an internal user which had a SSH key containing the flaw of openssl in 2008.
I hope, it clears up the things.
Note:
If you are going to image/backup the server before reinstalling, be very careful, how you do this. As @dfranke said, boot from a trusted medium to backup.
You shouldn't connect to other machines from a rooted server, as great rootkits are known to be able to spread through trusted sessions such as SSH.
The commandline can be changed if the process alters argv[0]. Try
ls -l /proc/[pid]/exe
From
man 5 proc
ps auxwf | less
gives you the "forest view" of processes which can show you what process launched this process (unless the rootkit is hiding it, or the app's parent has exited and it's been reparented to init).This would be mostly academic and probably just a timewaster, but
strings -n 10 /proc/[pid]/mem
might be fun to watch scroll past. You could alsoecho 0x7 > /proc/[pid]/coredump_filter
and use gdbgcore
to force a coredump with everything possible in it, but then the process dies, which could block further analysis.But definitely take Arenstar's advice. Back up data only, restore everything executable from backups, and start over. You should probably restore the website from backups as well, there could be malicious javascript added to every html or php file. If you're considering legal action, you'll want to just set the machine aside, unplug it from the net, and stop whatever it is you're doing until forensic experts have done their job.
Try "cat /proc/[process id]/cmdline" Though if it is a true rootkit it may modify the kernel to hide itself better.
You should reinstall, I agree. Have you tried escaping the characters in the path? Maybe one of those slashes is actually part of a filename and not a directory. At the very least you should use iptables to block outbound traffic to that host or IRC general until fixed. Check netstat as well.
I'd think by now you have reinstalled. Your justing wasting time trying to track the processes and do forensics as chances of anything legally developing from it would be very minute and chances of finding the hacker would be futile anyway. Unless it just interests you to research and reverse rootkits which can be fun:)