Advice on moving a machine room to a new location?

We have done a couple of moves with racks. Here's what I took away from the experience:

Before:

• label everything. Both ends. And the rack slot it's plugged into too.
• diagram everything.
• if you have important data, schedule a full backup to finish 6-12 hours before the move. At a minimum, validate the last full backup you have.
• plan to take everything out of the rack during the move. At a minimum take out half the mass, from the top down -- this means leave the top empty Don't expect movers to move a top-heavy rack. In a rack with a UPS at the base that means you are taking out probably 2/3 of the servers. Anything which is left racked MUST have rails and be secure in the rails (less than 0.5mm movement, and tolerant of any orientation -- including upside down and face down -- was our margin). We've moved full racks, and half-full racks, and the racks always need to be tilted/rotated, and it is downright scary to watch your livelyhood being dangled over a concrete loading dock.
• in our experience, front and rear doors are more decorative than load-bearing and leaving them on can make moving the racks very awkward because all the good hand-holding points are unavailable. Plan to remove the doors during the actual move. This forces the movers to lift, pull, push, and tug the rack by its strongest parts. Usually side panels can stay on, but be prepared to remove them if the movers think having them gone would help.
• hire someone with an insured truck to to the transportation. DO NOT do it yourself. You might think you are insured, but your insurance company will probably think otherwise. The second last thing you want is to be involved in a three-way blamestorm between your employer's insurance company and your insurance company.
• hire someone insured to physically pick up, move, put down, and (during an accident) drop your gear for you. DO NOT do it yourself. The last thing you want to do is hurt yourself or be involved in a co-worker's injury.
• make sure your insured someones specialize in high-tech moves and can supply blankets, anti-static bubble-wrap, and if necessary crates.

During the move:

• backups are good, right?
• do the wire disconnect yourself. Pull ALL wires that are disconnectable. Each wire should be labelled and go into a box that itself is well labelled.
• let the strong, well insured guys do the server extractions from the rack. Supervise the wrapping personally.
• stay out of the strong, well-insured guys' way when they do the loading and unloading.
• supervise the unwrapping and re-insertion into the racks. If there is ANY question about a server's condition, mention it, and note that it was mentioned.
• If the job is taking a while, have coffee and/or doughnuts available for the guys. If appropriate (and if they've done a good job), cold beer for after the job goes well too.

Physical bring-up:

• double your time estimate for getting things back on track.
• have a build-up plan. A good build up plan includes a checklist of all services, servers, and a test plan.
• validate, validate, validate.
• check any servers which may have been damaged in transit as noted above.
• don't let management cheap out when it comes to food if you are working through meal times. Hungry techs make mistakes. If the bring-up is going to be long and complicated, schedule time to get out of the room for 90 minutes to go get a real meal.

After:

• a cold beer for you may be appropriate, too.

I'm with Kevin on unracking and disconnecting. Racks that I've worked with that are built to ship loaded typically have additional bracing and must be attached to a fixed base (like a pallet) during shipping. Usually the servers have some kind of locking mechanism, too. If you're not moving something that isn't built to be moved loaded don't.

I would add this to Kevin's answer:

Part of documenting is knowing why the various connections and cables are there. You can make all the notes, take all the pictures, etc, that you want to, but at the end of the day if you don't understand why something is there you're running a risk of not knowing what to test to establish, after the move, that you "got everything".

This is a golden opportunity to, as you're documenting the "what's plugged into what" of the move, ask yourself "why is this plugged into this"?

I've had too many cases of the "I'll just patch this here and temporarily change a VLAN assignment" that turns into a semi-permanent configuration that no one documented come back to bite me when moving equipment years later.

Before any equiment goes into a personal vehicle, find out how the company's insurance is handling claims in the event of damage during transport.

Back everything up before you begin moving. Assume that all the equipment will be destroyed during the move because (if you're driving on public roads, at least) it could be. It goes w/o saying that the backup media should not move with the infrastructure gear!

Extract the individual components and move the racks empty. This also has the advantage of allowing you to reconnect things in an optimal configuration and get your wiring exactly the way you want it. Just be sure to document everything before, during and after. Take digital pictures to help.

We relocated two racks of servers about 60% full about 3 years ago.

It was the most frightening thing I have witnessed in a long time.

Everything was OK in the end, but the movers had to tilt the rack to get it out of the door at our old location. They were very big and very strong, and lucky for me the rack did not drop -- but I can tell you, I was stressing just watching it.

If you must to move the racks full, make certain that you can roll the racks through the doors with out having to tilt them at every door that you will encounter during the move. if you have to tilt, the forget it and start unloading servers :-)

I would personally advise on either emptying the racks first, or like Kevin suggested, getting new racks at your new location. This would also perhaps allow you the flexibility to move the systems over in stages, rather than all at once. Of course that might not work given your specific environment.

Buy an awesome label maker, and label everything like you're going insane. Then take it apart and move it piece by piece.

That is all.

You might also consider a partial move. For example, you might move backup equipment first - if it's a live backup, you might even redirect users to the backup systems while the primary systems are being moved.

You might also start the move with one team's systems, or part of several teams' systems first. Fix the problems that those systems experience while the team can still work with their machines in the old room.

before you move the equipment, prepare what you can ahead of time. if you want a tidy rack, plan where each server will go and cut new network cables to length and have them labeled. install any extra switches and pdu's as well. this way when moving you only have to take power cords and servers.

when moving, if you are moving production equipment, domain controllers, email servers, terminal servers or anything that users generally expec 24x7x365 uptime on, get those shutdown last and booted up first. Move equipment later in the night and if you have a lot to move, get a couple extra hands (friends will work for pizza) and borrow/rent a van. If your boss is paranoid about security, contact the local sheriff about a police escort.

get your business critical equipment up first and tested. secondary equipment like backup servers or backup dc's can wait a bit. doing work overnight gives time to bring things up and tested without interfering with the

these are a few things i learned moving 2 racks worth of servers in one night.

Move it empty (or get new ones installed and cabled in the new location first). Servers are staggeringly heavy. Even if you wanted to I doubt you'd be able to move 42U of densely-packed electronics in one go.

I am all for the already existing answers about labeling and unracking. The other thing I would add is if it is possible don't move all 5 racks of gear on one day. That way you only have to trouble shoot one rack worth of gear in a single day.

I just did this a few months ago. Here's my tips:

1. Get a ton of sleep. I ended up working 32 out of 36 hours starting at 8am Friday and ending at 10pm Saturday.
2. We were fortunate enough to have most everything setup in an HA cluster, so we could move half the equipment to the new DC at our leisure during the day, switch DNS to the new DC, then move the remainder of the equipment as DNS is propogating.
3. If you're going to change IP's, crank down the TTL on your domain ahead of time so propagation happens faster.
4. Make a very detailed checklist of steps laid out in chronological order. You will be physically and mentally exhausted, having a checklist leaves you little room for mistakes.
5. The most important part: have backups and service plans on everything. We had no bumps or slips during our move, but we lost 2 hard drives, one stick of RAM, and one DRAC card in the move. Probably vibration from being moved, but it could have been simply power cycling it all.

I did nothing but prep work for two weeks prior, and it paid off big dividends.