Our building is located approx. 100 meters from the explosive charges. They happen several times per day, and really shake the entire building a lot. This is going to go on for many days and the blasts are supposed to get stronger.
Our server rooms are nothing fancy; one of them has all the racks on hard concrete while the other one has a raised floor (the one which allows the cables to go beneath it).
Does anyone have any tips, countermeasures or best practices for us?
Currently we are thinking of the following countermeasures:
- Daily report of the server rooms status lights (HD lights, power supplies and so on).
- Nightly check disk scan on the most important servers
- Order in extra supply of spare harddrives
Edit: Many good answers here! However one needs to be accepted. The highest voted answer at the time of this edit will get their answer accepted.
If I were you, I'd call the company's insurance company, and have them place out an accelerometer.
This way - the insurance company will know that you're not the one shaking the disks, and the insurance company will know for sure when every blast has gone off, in case your SAN dies at the exact same moment.
(We did this.)
Now would be the time to make sure you've verified your backup solution. All the replacement hardware in the world won't save you if your backups are corrupt or have otherwise been rendered useless.
In addition to all the other excellent suggestions (particularly off-site backups) you should consider dust proofiing your room to the extent practical: Weather stripping around the doors, tape around the windows, etc.
If you have external air intakes plan on changing the filters when the blasting is over.
All that said, I wouldn't waste time/resources on nightly disk scans, or money on spare hard drives (maybe buy one or two for your most critical machines, but remember that you can overnight drives if needed).
Server equipment is surprisingly robust, and will continue to operate for years under some pretty awful conditions. Chances are you'll be fine.
Mount your server racks on ISO-Base platforms. This will isolate your server rack from the ground shocks and allow your systems to function even in the event of movement.
It's not easy, but is effective. Especially in earthquake zones.
To see the ISO-Base in action, check out this video showing the effects in a server rack during an earthquake simulation. One rack is mounted on an ISO-Base, the second rack is not.
Power: If you lost power, will all of the servers, storage, san (etc) in your rack reboot on their own? Or will they come up only after you press the 'On' button?
This for two reasons:
You might also suggest to your users to turn off, and unplug computers each evening.
Most people talk about backup - not many talk about the restore. Make sure that restoring from backup is as easy as possible. In particular if time is critical.
If you have a co-location replication, switch those to be the masters (if timing permits) since the ones in your building are much more vulnerable. They are also more likely to go down during work hours (evidently).
I also advise working from home.
Play heavy metal to replace the hard rock as it explodes. Hard rock is so passé anyway. ;)
More seriously, maybe there's a DJ gear shop around the corner where you can pick up some shock absorpbtion/isolation stuff. Many (dance music) DJ's have a similarly hard time when they're playing records (i.e. keeping a tiny needle in a tiny spinning groove, somewhat comparable to sensitive equipment like disk drives) at extreme volume while the crowds around them are jumping and stamping on the floor in sync (the "in sync" is important; if you have never heard of the Tacoma Narrows Bridge incident, you should consider watching this clip). Rip a few trash bags open to provide additional protection against dust.
I'm surprised no-one has mentioned the process side of things.
This is the perfect opportunity to go over what you have planned for business continuity. What is the plan if you have to move offices for a day or a week? Do you have up-to-date plans on drawings and which systems have priority for restoration? Is management briefed that you do have a plan and aware of
The acceleration from the blasts causing havoc on the server room is probably the least of your worries. Your utilities could be at much higher risk unless you are self sufficient with on-site power and robust connectivity (assuming you are not self-contained and only supporting local staff).
If there is a water main or power or internet access failure, can you survive that? Have you called your internet provider to see if they are aware of the blasting and have prepped to restore service through an alternate route if your utilities are interrupted. You'll know your specifics better than we can guess, but you should have a list of everything you need to function and addressing "What if this goes away unexpectedly?" for each.
Just going over this in your head / on paper will help you know if you have any weaknesses that need work later and perhaps communicate this up the chain if your organization doesn't have anything written up. Start with a two page, executive summary - just a FYI so that everyone knows what you're doing.
Yes - getting a few extra hard drives / spare parts on hand is good, but I would be more worried about the things I can't see or don't directly control.
The real benefit of this process exercise is a reality check for your current monitoring system. Once you've planned out some basic scenarios, you'll be better prepared for the unexpected. Having a short summary of what you expect to survive and what you don't will come in very handy no matter why you suffer an outage, and also assist in driving your efforts to improve monitoring 24/7 rather than when the foundation starts shaking.
I used to run computers on a mine site that was being reclaimed. As the reclamation got closer and closer to the office, the data room would shake almost daily at the end of the day.
My counter measures included moving the racks on top of some hard rubber feet designed for vibration dampening. They were made by Manson and I just estimated the weight of my rack and bought the appropriate feet. This seemed to fix any movement within the rack. The tough part was lifting the rack gently enough to slide the feet under.
Also, as other had mentioned, I had backups that I tested and kept off site.
Those servers lasted 3 years with constant shaking an no hardware related crashes. The desktops in the office did not fair so well as 2 of them had catastrophic hard drive failures.
I have not seen this mentioned about backups yet, but make sure you take them off site! You may also want to make a backup of switch, firewall, etc. configurations.