How to detect a hidden process in linux?

You cant trust any system tools you have on the machine. Rootkits will replace ps, netstat, ls, and more to hide their presence. You should take the machine offline, take out its hard drive, make a forensic copy (think dd) then work from that on a seoncdary machine to scan for the rootkit.

If you insist on working on the live machine (which is usually futile) then you can try downloading a rescue distribution on CD (its very important the copy be read-only) and use its copies of ps,lsmod, etc.

Even this might fail as a rootkit can install kernel modules to hide entries in /proc where tools like ps normally operate on.

Good luck!

The problem with well constructed rootkit is that they modify your system's command; like ps and top to not show the processes of the rootkit and ls to not show files of the rootkit.

So what you'll need to do is to get these command possibly from source or in binairies form. (Be sure to be well signed). But the trick of a root kit (I've seen it) is that they maybe corrupted your compiler too. So when the compiler knows he is compiling ls or ps or any command he infectet them as well.

When I saw this problem I said fine lets recompile gcc, but no what I need to have to compile gcc...the infecte gcc....so when he knows he is compiling itself he infect it so it can infect the command.

You will say that this come big and difficult to detect, yes but rare are the root kit that are so bulletproof I just gave youthe worse case.

Seriously, if you are sure that there is a root kit in your server, reinstall it!

The best way to know whether your server has been "rooted" is to be running a host-based intrusion detection system (HIDS). Unfortunately, if you're not running a HIDS now, then it's too late to install one. The proper time to install a HIDS is when the server is first installed, and before it is put onto a network.

Briefly, most HIDS work by computing cryptographic hashes of all system binaries, and storing those hashes (along with numerous other file statistics) into a database, called the baseline database. Then, periodically, the HIDS rescans your system, comparing all files in its baseline database to the actual system files.

Yes, of course, it is possible for a rootkit to modify your baseline database, which is why you need to take a copy of that database and store it separately of the server before you put the server online. Then, if you suspect you are "rooted" (and you suspect your baseline database was also tampered with), you can boot your system from the install media, restore the known-good database from your backup, and then run a scan against the known-good. It is much more likely, however, that a rootkit will not anticipate having to defeat your particular HIDS, and so you will receive a notification from the HIDS that system files have changed, indicating a probable system intrusion.

Since you were not running a HIDS, you have no quick way to determine for certain whether you have been rooted, or what system files have been modified. You could spend a whole lot of time comparing your system files to known-good files pulled from known-good installation media, but that time is most likely better spent reinstalling your system from that media. If you want to investigate how you were rooted after the fact, the best course is to take an image of your system before you wipe it and reinstall.

Please refer to an earlier post made

Pain removing a perl rootkit

Its really important you read this..

As to answer your questions..

I usually run an IDS/IPS (intrusion detection/protection system) like snort.. it does a great job against foul play, and i have seen it doing a GREAT job in action..

I also use Syslog Servers to keep log messages off production servers, so you can backtrace problems and changes / being rootkit'd

Im also often using management tools such as cacti, which graph cpu,memory,disk,network usage and report if anything is out of the ordinary..

Being rootkit'd is a serious problem, definitely try to work out the cause..

Hope this helps.. :D

The random high ports are the ephemeral ports, which indicates that you likely have one or more programs connection outwards from this system.

If it isn't rooted, then netstat -np | grep -v ^unix may give you a hint as to which program(s) are generating the traffic.

You may also be able to scan the traffic from a nearby system using tcpdump to dump packets originating from the system you believe is rooted. If the problem program is not running as root, you can do this from the infected system.

There are several things you can do to avoid getting rooted:

• Keep your system patched, especially the kernel. Monitor the reasons for the kernel release to determine the attack vectors.
• Only install and use required programs. Do as minimal install as possible.
• Run as little as possible as root. Many programs will change to unprivileged userids once they have completed the setup requiring root privilege.

EDIT: If you are rooted, then using statically linked ps and ls programs may indicate a difference between the running programs the two tools find. Differences in the list can also arise as short lived programs die off.

The real fix for a system that may be rooted is to reinstall it from secure sources. Like the installation CDs. Then restore your data only from backup. Any binaries or scripts in your backup may have been compromised and saved into your backup.

To find the root kit on a running system, one way to do it is to compile a kernel module designed to detect rootkits. Compile it on a different machine that is running the same OS version. Then copy it over and insmod it.

A rootkit detector will have tools built into the kernel module to dump the running process table, verify the syscall table (to look for syscall intercepts) and other features. It may have scripts that will take the output from the module and compare that to the output from ps, netstat, ls, etc. Or it may be able to scan the memory in kernel space for known rootkit signatures and report.

This is like mice: if you see one, there's a hundred living in there. If you see signs of a compromise, you've got to assume the whole system is compromised. That's why everyone is suggesting that you reinstall the system rather than spending a lot of time looking. I've encountered several situations where a machine was compromised and sysadmins thought they'd cleaned up the mess, only to regret it later.

It is very common for rootkits to replace system binaries such as ps, top, netstat. And it's also common to trojan ssh. So, looking for checksum oddities in those files is a prime approach. If you're on an rpm-based system, rpm -V is usually a good tool, or dpkg-verify on Debian/Ubuntu. Or you can check the checksums directly (but watch out of prelink, which changes binaries on the fly for speed reasons). This isn't reliable, but many script-kiddie attacks don't cover these traces. (In other words, if you find something, good. If you don't find something, it doesn't prove you're clean.)

Other things to look for: ports shown open from an external nmap which do not look open via netstat on the machine, and pids in /proc which don't show up in ps. And if you happen to have remote syslog enabled, look for ssh logins which don't have records in the lastlog.

Grab yourself a copy of RKhunter and chkrootkit. They are usually pretty decent at helping find things that shouldn't be there.

It's always good to run mod_security as well on your Apache layer along with a firewall. Usually they'll find an outdated webapp or script and enhance access from there with Perl shell scripts and other fun things.

ps auxfww is usually great to find processes that don't belong.

Also: netstat -anp to see what is listening on certain ports.

Again these are good if you haven't been rooted, just compromised.