We run 10 servers, mostly cheap requisitioned desktop machines, all running CentOS 5.1 and Xen. All these 10 servers do is run Xen virtual machines. Someone switched off the power supply to the server cupboard and a couple of machines have fried harddisks and will no longer boot. No worries, all the virtual machine disk images have DRBD mirrored copies on different servers, I just need to start those up while I work on getting those two machines back up.
I've replaced the boot drive on each of them and re-installed CentOS, Xen and DRBD. However, after a reboot, the severs can all see their local DRBD devices just fine but report a status of "WFConnection" - waiting for connection - for the remote link. Each device reports as "Secondary/Unknown". I've checked:
- Each server has iptables configured correctly to let DRBD traffic through - this is definite, the iptables config file is from a central repository and is identical to what it was before the machines crashed.
- It's not a DNS problem, as each server has a fixed IP address and DRBD.conf uses those IP addresses directly, so DNS isn't even used. I've made sure the new installs use the same fixed IP addresses and hostnames as the original servers.
- Each server can ping the other one on all IP adresses used, no problem there. The server are all connected to the same switch.
Does anyone know why DRBD is still refusing to connect?