John 'Shuey' Schuepbach's questions

Due to hurricane Matthew, our company shutdown all servers for two days. One of the servers was an ESXi host with an attached HP StorageWorks MSA60.

When we powered things back up today and logged into the vSphere client, we noticed that none of our guest VMs are available (they're all listed as "inaccessible"). And when I look at the hardware status in vSphere, the array controller and all attached drives appear as "Normal", but the drives all show up as "unconfigured disk".

We rebooted the server and tried going into the RAID config utility to see what things look like from there, but we received the following message:

An invalid drive movement was reported during POST. Modifications to the array configuration following an invalid drive movement will result in loss of old configuration information and contents of the original logical drives

Needless to say, we're very confused by this because nothing was "moved"; nothing changed. We simply powered up the MSA and the server, and have been having this issue ever since.

The MSA is attached via a single SAS cable, and the drives are labeled with stickers, so I know the drives weren't moved or switched around:

---------------------
| 01 | 04 | 07 | 10 |
---------------------
| 02 | 05 | 08 | 11 |
---------------------
| 03 | 06 | 09 | 12 |
---------------------

At the moment, I don't know what make and model the drives are, but they are all 1TB SAS drives.

I have two main questions/concerns:

1. Since we did nothing more than power the devices off and back on, what could've caused this to happen? I of course have the option to rebuild the array and start over, but I'm leery about the possibility of this happening again (especially since I have no idea what caused it).

2. Is there a snowball's chance in hell that I can recover our array and guest VMs, instead of having to rebuild everything and restore our VM backups?

So Microsoft claims that "you cannot use a network shared folder as a backup target for a system state backup", but I've seen plenty of posts where people indicate that they are able to do this from a command prompt using wbadmin.

My end goal:

I am not concerned about backing up any of our domain controllers, because if one dies, I'll just spin up a new domain controller and let the remaining DCs replicate to it. I am concerned however about at least making sure that I have a backup of AD, in case our entire AD infrastructure were to get hosed and needed to be restored from a backup.

Here’s what I’ve done so far to accomplish the goal:

From my PDCe, I ran a successful backup to a network share using the following command:

wbadmin start systemstatebackup -backuptarget:\srv-backup\b$\srv-dc1

I then created a scheduled backup like this:

wbadmin enable backup -addtarget:\srv-backup\b$\srv-dc1 -systemstate -schedule:03:00

I verified the following day that the scheduled backup completed successfully.

So here are new questions:

1. How do I properly backup AD? Is my current method correct?

2. If my current backup method will only yield ONE backup at any given time (because it’s backing up to a network share and it will overwrite the previous backup each night), should I look into getting local storage to push the backups to (so I can have multiple backups), or should I just do backups of my other two DCs in the same manner; to a network share (staggering the schedules of course - then I’ll at least have one or more daily backups that I can depend on)?

3. I've read in another thread in the community where someone said to "backup the NTDS folder from C:\Windows", but I'm assuming that is unnecessary since it gets backed up during the systemstate backup - is that correct?