nmajoros's questions

Two weeks ago, two of our four RHEL7 VMs running on same Openstack infrastructure rebooted unprompted and unnoticed. May be externally triggered.

Ever since, all services relying on DNS are unreliable on these two servers, while alright on the unrebooted ones.

After diagnosing, it turns out the affected ones have an average dig time of 5 seconds vs 0, with possible time out. Pinging/ICMP is not available but strangely enough no other IPs seem to have a problem being reached.

dig @1.2.3.4 example.corp  +short  

All four VMs are running same OS, same /etc/hosts, same resolv.conf, same nsswitch.conf . All four VMs traceroute the DNS server through the same gateway.

What else can I use to debug the issue?

We are conducting bi-yearly resiliency test on a 4 node VCS cluster. With two applications running on nodes 1-3 and 2-4 resp. in active-standby mode.

When making a manual switch over, or a graceful shutdown on one node, application will nicely switch over to the other node.

When we turn off or reset a node however, it seems the absence of a handoff by the affected node triggers a reboot of the other cluster nodes. What's more, in case of a turn off, the remaining nodes do restart, but fail to join the cluster. When the killed server is resumed, all join again.

This is totally defeating the purpose of a cluster, obviously. Our vendor who provided the applications and the cluster software (with the hardware) proposes that such a case is not realistic and that servers always hand off nicely when going down.

We have no specialization in proprietary cluster technologies, so while we assume their statement is incorrect, we don't know what is possibly going wrong. I suspect however any commercially successful cluster software can handle these situations, but our implementation suffers configuration errors.

Any clue would be appreciated.