On an old HP SE316M1R2 (Same machine as the much more common ProLiant DL 160 G6) I have a 4x450GB Raid 5 array handled by the stock P410 Smart Array controller, fw ver. 8.40.30.00. Drives are genuine 10k SAS 450gb 2.5.
One of the drives (Bay 1) is marked as "predictive failure" so I purchased a spare one.
First try was to shutdown and remove the old drive, substitute with new one, turn on. The system did not find the boot drive. I've put the broken drive back in the first bay and the server booted again.
Second try was with the broken drive in place (bay 1) and new one in bay 7. Drive in bay 7 is recognized as raid 0 one drive healty array. I deleted it from logical drives. In the controller's firmware utility I was unable to find any way to either mark bay 1 drive as failed, set bay 7 drive as replacement or set bay 7 drive as hot spare. Even rebooting after deleting the old logical config from bay 7 drive.
I then got into the CLI command interface as last resort and I was unable, typing help, to understand how to work this issue out. No commands seems capable of removing or adding or swapping drives into existing arrays.
I can't belive I should delete the volume and recreate in order to get rid of it. I must be missing something, so:
What is the best practice to swap drives in this scenario?
Note: VMware ESXi 5.5 is installed in this machine.
Edit: I probably figured out what's happening. @ 1st try the drive newly inserted in bay 1 already had its own logical drive configuration. That's why the array was not recognized and the system didn't boot. @ 2nd try I've wiped its configuration. Now that the drive is blank is it possible that if I go on through 1st try again the controller will read array configuration from other drives and boot, and start backgrounded array rebuild? Will update again when I'll be able to schedule another maintenance window for this machine.
On SmartArray controllers the simpliest way is to swap hot-swap the drives online, this will start immediate rebuild. You will be able to monitor the rebuild process with hpssacli/hpacucli tool.
You can install these tools on ESXi, it's available as VIB on HPE site.
Be aware that it's common for RAID5 arrays to discover another failed drive during the rebuild and lose the data. Hope you have backups.