Pull network or power? (for contianing a rooted server)

RAM forensics (e.g. /dev/shm) can be helpful.

But I prefer unplugging the power cable (but try to log-in and rsync /proc right before).

The reasons for going for the power cable are:

1. When you do forensics in a hacked system, you are "stepping all over the crime scene"
2. The root kit keeps running - not so hard for the malicious to execute something (e.g. system wipe-out) on Network Link Down event.

Kyle Rankin gave a nice Intro to Forensics talk - there he recommends pulling the power cable.

Disconnect the network. The attacker can't pull any additional information if there's no network connection. It's very hard (read: impossible) to do any forensics without power.

In this day and age, it could a Virtual Machine so either method is easy and can even be done remotely. (Virtual machines give rise to the possibility of using snapshots, too, of course)

I'd suggest disconnecting from the network as an automatic first step, because this gives you time to ponder what the next step should be, whether that next step is yanking out the power cord or something else. If you know your corporate "security response procedures" won't allow you to spend time digging through the machine in detail then preserving the contents of RAM might not be that important.

I'd suggest that getting "separating the victim from it's muggers" done in any way is more important than the how, so both approaches are valid. I'd have no problems with pulling the powercord myself, lets put it that way.

This isn't an either/or situation. You generally want to do both -- you want to perform certain forensics (dump of running processes, listening sockets, files in /tmp, etc.) on a system that's been removed from the network, and then perform your remaining diagnostics from a safe environment (i.e. a live CD). There are situations where neither approach is the right one, though, and you need to think about and understand what those might be in your organization.

If you're facing an APT, then your best option is to set up a honeypot and thoroughly investigate all traffic that flows into and out of it, in addition to monitoring the server.

The measure of going through memory is so expensive in terms of time and effort that it's usually not worthwhile unless you've tried every other method, and if you determine that it's worthwhile, it's generally best setting up a honeypot that allows you to easily dump the memory and system state to another machine on the fly so you can do analysis with less threat of being detected while the machine is up and running.

I had one situation where the attacker kept everything in memory to the degree that, except for logs, the machine looked exactly like its image once powered off and back on. They would then hack back in and start using it again because the vulnerability was still there - they didn't need to leave any backdoors for themselves. A memory evaluation could have helped here, but watching the traffic was enough in this case to identify the vulnerability quickly.

Therefore:

The only reason to avoid pulling the power and doing offline disk evaluation is if you're going to go through the pain of doing a thorough memory analysis of the threat while it's in place and operating. If you've gotten to the point where this is necessary, then there is no reason to pull either plug.

If you're not doing a memory analysis, then pulling the power plug is your best bet - pulling the ethernet (or using a shutdown command) is only going to give the attacker's software advance notice - which does matter occasionally.

So:

Pull them both, unless you're doing a memory analysis, in which case, don't pull either.

Before you do anything, figure out if you need to preserve evidence for an eventual prosecution. Evidence handling is a very tricky subject and not for the faintly-trained. Once you've answered that, then the trained computer forensics person can take it from there.

No need to power off the server. You can just disable the connectivity from the border gateway / router. One firewall rule will be enough to discard any further sent/received packet.

The answer depends largely on what you mean by "rooted". Pulling the network lead is usually a good idea, particually if you believe this was an active human attacker looking for sensitive information.

Pulling the power is much harder to answer. If you feel there's malicious code running which may cover its tracks, do so. However if you feel there may be evidence of a break-in still in the memory, leave it running.

Overall, it depends if you're dealing with malicious code, disgrunteled staff, or someone with a specific target in mind.

If erring on the side of caution, pull the power. You'll lose a small amount of potential evidence, but may prevent the massive removal of other evidence by automated code.

-- Then again, if you feel the machine has been compromised and you know it has no sensitive data on it, you're very confident of your network security elsewhere and understand the consequences of doing so, perhaps get a forensic boffin in asap and see if you can trace or understand the attack and go from there. That's not normally a good idea, though

My answer was before the edit, with the 5 assumptions.
My assumption was that if your server got rooted, you're dealing with a sophisticated, targeted attacker, and that you have no way of knowing when and where the attack came from. (Since the machine was rooted, obviously you cannot trust anything it tells you. However, you might have some off-box information, e.g. IDS...)
I assumed also that you have interest in dealing with your attacker, and not just waving him off like a bothersome fly. If the assumed attacker is a script kiddie, this would be different. I also assumed that, since the attacker is targeted and sophisticated, it's likely that they have custom software running on your machine, that can respond to your actions on it...

Neither.
Check out https://security.stackexchange.com/q/181/33, either way its a bad idea.
It's like giving a couple of bandaids to the black knight... too little, too late, it just ain't gonna help much.

For all those who feel that this answer is too far out there, or just not "security-conscious" enough - you need to realize that it's already too late.
It's not your server anymore, even if you completely disconnect it. Even if you shut down, run all your AVs and whatever - you don't own it anymore, they do.
That's kinda the definition of "rooted".

Now, assuming that they own it, and can hide their ownership from you, you need to consider - what will disconnecting it acheive?
The only thing it will acheive - since it will continue to be rooted regardless - is to let your adversary know you're on to them. Which simply puts their guards up, and gets them to start cleaning up after themselves (if they haven't already), which will just kill any hope of forensics.
You're still not protecting that server, or any of its data. How long has it been rooted? How would you know?

What you're better off doing:

• Leave the server up and running...
• Isolate it from the rest of your internal network, other servers, etc - but best to plug in some kind of simulation, so it seems connected.
• Quietly start realtime / network forensics, run traces, etc.
• Try and figure out the extent and length of the damage (obviously its different if it happend 2 hours ago, or 2 months ago).
• Continue from there... Only after mitigating the actual damage, go ahead and clean it up.
• Of course don't forget to investigate the root causes, and enforce whatever controls to ensure it doesnt happen again...

Having reviewed your assumptions do both. Use a CD/DVD based media to dump the current state. Primarily you will want to be able detemine how you were compromised. You may also want to attempt to recover user data not on your backup images. Yo

Then rebuild the system from the latest backup media that is not contaminated. Verify this with checksums if possible. Alternatively, reinstall the software from installation media, and reconfigure. Reconfiguration should be automatic if you have been using Puppet or cfEngine to configure your server.

Reload user data and scan for root kit components. Verify you have no setuid or setgid programs in the data directories.

Determine and close off the method of access used to infect the system. Stage reactivation of the server allowing time to verify applications are running as expected. Carefully monitor for new infection attempts.

This all assumes the infection is at root level. Web infections can be done by altering the code run by the web server. Allowing the web server write access to its code may make this more easily done. You may still want to treat this case as if the root account has been compromised.

If root on one system has been compromised on one system, you may have more systems compromised. Carefully consider which systems can be accessed without a password from the infected system.