I'm trying to compile a procedure for when ever there is a suspicion, perhaps from netflow data or something like that, that a server could be compromised. The logic being, that you should think about these things coolly and calmly before they happen for real and the pressure is on and the adrenaline flowing.
Can anyone recommend some good resources I should familiarise myself with to help me draft a sensible procedure?
I'm looking for good information on what to do to confirm suspicious, as well as what to do when a hack is discovered.
You can run something like chkrootkit on a schedule. http://www.chkrootkit.org/ Same for rkhunter.
Look for suspicious processes and shutdown unnecessary daemons.
If this is an rpm-based system, you can look at the output of rpm verify (
rpm -vVa) to look for changes in installed packages.There's always Tripwire... http://tripwire.org/
Assuming that you're explicitly not addressing the issues of preventing a compromise, nor the issues of recovering from a compromise....
Before it happens
You need to have secure logging in place - i.e. log data should be immediately published to a separate box for recording.
You should have a host based IDS to detect unauthorized changes - again with data storage off the box - such as tripwire / LIDS
You also need to plan for what you're going to do immediately you suspect a compromise - have you got a seperate unit you can swap in? If its a straight copy, then it will have the same vulnerabilities as the box it is replacing. Can it be configured to provide a reduced service with better security (e.g. a webserver with a read-only filesystem and bare-bones content).
Decide criteria for involving law enforcement. If you may be involving them - and they are likely to be interested - go speak to them in advance and ask how you can make their life easier.
Get agreement from all the stakeholders to the planned response.
Detection
In addition to basic anomoly detection, you should be checking the output of the intrusion detection system, and running rootkit checks regularly, also running frequent port scans against the box. Your routine anomoly checking should include log analysis.
While the methods described above are of value where the system is modified by the attacker, they do not address the problem of information disclosure. AFAIK the only sensible way to address this is via honeypot data (e.g. email addresses, user accounts).
when it happens
Pull the plug out. Seriously. A system shutdown may make significant changes to the system. You want it disconnected from any other devices on the network as soon as possible - but you need to preserve as much as possible about the state of the system.
If you're going to involve law enforcement - let them know before you do anything else.
If you want to investigate yourself, boot up the system from a USB / CD - NOT from the installed OS.
You won't determine this just by looking at netflow output, since that is not nearly detailed enough to tell.
Instead, I would suggest using netflow suspicions as a starting point: look at it for some time, try to see if it is trending up or down, and if you suspect there may be service outages caused by it, pull the plug on the affected machine(s).
You do not perform forensics on live boxes, ever; if the suspicion becomes acute enough, you pull the box off the network, and then examine the fallout at your leisure.