I'm planning on compiling my own kernel, which I've done many times before but as always there's still a chance this could go wrong and the machine could fail to reboot.
This machine is within a remote data center to which I have no physical access and thus it would cost me a great amount to have someone to physically go to the server to fix it should the machine fail to boot correctly.
Is there a way I can set grub to use a new kernel for a single boot only but fall back to the old kernel upon an additional reboot. So I could use my remote power management console to reboot the machine if it failed to boot correctly with the new kernel, it would then fall back to the old kernel.
I know this is possible with lilo using the lilo -r kernel
command, is there a grub equivalent?
To quote a bit from the GRUB Manual which adds a bit of extra checking for katriel's answer.
You could try using the "fallback" directive in /boot/grub/menu.lst. You will need to specify a fallback kernel or kernels, and use the savedefault directive in the kernel stanzas to be used as backups, for example (taken from the grub manual):
This will cause the A kernel to boot, and if any failure occurs (kernel not found or kernel panic), the first and then the second fallback will be used (as specified in the fallback directive)
Please note that the manual specifies a missing kernel or kernel panic, it says nothing about a network card not working. Which in your case can be as bad as a kernel panic