Problem: The SSDs in our servers require a power cycle (i.e. off/on, not reset/warm reboot) after a firmware update.
Thoughts: Using 'ipmitool chassis power cycle' I can cycle the server's power. However this would cut the power while the system is still running, filesystems are mounted, etc.
What I basically want is a delayed power cycle so the system has a chance to halt. But I guess that would have to be implemented on the server's IPMI board, so it's not really an option.
My initial idea was to dynamically create a ramdisk containing the tool and libs and somehow integrate that into the halt process. I saw there's a /etc/init.d/halt, so that would be my starting point. Although I believe the kernel at some point in the shutdown process starts to kill off remaining processes. So I'm not even sure if that's a viable way.
Question: What would be the best way to execute ipmitool (or any other command), after the system has halted and all regular filesystems are unmounted?
I would take it down to single user mode, unmount any non-OS filesystems and remount any existing ones as readonly. Then run your firmware update tool and 'ipmitool chassis power cycle'.
It's not pretty or clever but should be safe enough
Do a normal full shutdown and power-off; count to ten and then power the machine back on with wake-on-lan or IMPI to the BMC (DRAC in the case of Dell?). If you want to keep it entirely local to the machine, some BIOSes will let you specify a wake-up/power-on time via ACPI - see http://www.mythtv.org/wiki/ACPI_Wakeup . E.g. set power-up time to (now) + (normal shutdown duration) + (a small safety margin), then run shutdown.
I've no idea if that's actually possible with server-class hardware - please let us know if you try it.
You're looking for a watchdog. A timer that is activated by your software and resets the computer if the timer expires without being refreshed. For this you either need a BIOS with a supporting timer to turn the machine on, an external IPMI manager, or the good fortune for the IPMI power cycle support to work on your machine directly.
As a starting point, check out IPMI util which also includes the tool with a "power cycle" reset.
If you can set the watchdog timer active, then establish a runlevel that starts the timer with a timeout function of hardware poweroff. The last effect of your shutdown-without-ACPI-off runlevel should be to kill the timer update process.
At startup (instead of shutdown).
Using a network boot.
You need to power cycle the physical drives, so I suggest you reboot the machine, telling it to boot from the network, once done, you power cycle the drives, and then reboot from the drives.