In case a Linux server was exposed to the internet with extreme low security policy (r/w anonymous Samba folders, Firebird database server with default admin password, no firewall, etc.) for a week, then how do I make sure the system is not compromised without full formatting&reinstalling, accessing it only remotely via SSH?
Normally I'd suggest a local check with a tool such as chkrootkit but if the only way to run the check is to do so remotely, then I would recommend that you try Rootkit Hunter instead.
Rookit Hunter checks for rootkits and other such activity by running tests such as the following (see Project Information for more details):
I want to add that as others have said, the only sure way to ensure there has been no tampering with your service is to rebuild it. These tools work well, but they are not a 100% guarantee of success.
OSSEC checks for rootkits and detects suspicious activity.
I know this answer isn't what you want to hear but here we go anyways. There are some tools that can check the system the best way to ensure the system is clean is to wipe the server and rebuild. I would do the following:
Here are some resources I would start reading if you haven't already.
[link text][1] link text link text link text
[1]: http://www.sans.org/reading_room/whitepapers/linux/linux-rootkits-beginners-prevention-removal_901"Linux Rootkits Beginners"
Also not the answer you want but if there is a possibility that a system was rooted, it may be very difficult to be 100% sure the system is clean. Rootkits are designed to be difficult to detect. If you run the various root checkers and it checks out clean, then "most likely" your system is clean.
If security is a concern, I would consider rebuilding it as above poster just said or restoring it from good backups.
You really need to be proactive here. There's no reliable way to detect rootkits on machines, you thus need to prevent them getting there in the first place and find ways to detect them upon entry (e.g. via tripwire and locked down interfaces).
If you think a machine has been exploited in any way, you really need to reinstall - there's no guaranteed way to clean it up short of a reinstall. By far the safest option.
RKhunter, Tripwire etc are great, but really only of benefit if they were installed before the incident - this is because they are great for detecting whether key files have been changed. If you install RKHunter now and run it, it will detect the inclusion of many rootkits, but it won't detect any backdoors an attacker opened up in the OS or the applications you use.
For example, you could sneak onto a computer, create a new user, give them SSH and sudo permissions, and then clean up afterwards leaving a legitimate looking config in place, and no rootkits - then come back later and do your evil.
Best thing to do is look at what ports have services listening on them, then look at the configuration of all those services and make sure they are all legitimate. Then look at your firewall configuration and lock down ports that you don't need, both in and outbound. Then install RKHunter etc to see if some script-kiddie dropped a root kit in there messily.
To be frank, it is probably less work to do what JJ suggested and rebuild than making absolutely sure the computer hasn't been compromised. It's the data that is valuable, not the OS and config (apart from the man hours in setting that up).
You'll never be sure it wasn't cracked by someone smarter than you.
The first step should really be rkhunter/chkrootkit, however I've also had good luck in the past with the features that come built into certain package managers, for example 'rpmverify' which will go through all the packages on your system and check that the MD5Sums of the files they included don't differ from the files on disk.
Core binaries should really have identical MD5s to what's specified in the RPM or DPKG databases, so if they're different you know there's something weird going on.
The most effective way to determine whether your running system is compromised is to use Second Look. It will verify the kernel and all the running software in memory to make sure they are consistent with what the distribution vendor shipped. This is a far better approach than rkhunter, chkrootkit, etc, that look for artifacts of specific known infections. Second Look makes no assumptions about the integrity of the operating system, so you don't need to have used or installed it before an incident.
(Disclaimer: I am the lead developer of Second Look.)