I have a pair of CentOS Linux servers in each datacenter. They have failover within each datacenter, managed by heartbeat and DRBD (I know these are outdated tools, but they are stable, so there's no desire to change them).
They also have the capability for inter-datacenter switch as well, to make the east datacenter active while west becomes passive. But this is a manual engineering process, and that's okay.
The west datacenter is currently the active one, the east datacenter is passive.
serverA.west <-> serverB.west <-----------> serverA.east <-> serverB.east
ACTIVE DATA CENTER PASSIVE DATA CENTER
Servers can run mysqld and a Java application.
The Java application on this server should run only on the Primary host in the active datacenter (i.e. serverA.west). If another instance of the Java application starts on the Secondary host (serverB.west), or on either host in the passive datacenter, there's a risk of split-brain problems.
Today serverA.east rebooted, which caused heartbeat to flip over to serverB.east. Heartbeat then dutifully started the Java app on serverB.east, which we don't want to happen.
Heartbeat also started mysqld on serverB.east, which is correct, because MySQL replication should keep going, replicating the changes from the west datacenter continuously so the east DC is ready to take over when needed.
/etc/ha.d/haresources names the /etc/init.d scripts for mysqld and the Java application as the resources to start.
We want to allow heartbeat to manage the A/B pair in the passive datacenter. It should start mysqld on a failover, but not the Java app. But if the east datacenter is the active one, then heartbeat should start the Java app during a heartbeat-automated failover.
What's a good way to implement this?
What I am hoping for is something that takes one step to configure as we switch the active datacenter from west to east. Ideally, it should be mistake-proof, i.e. it should be guaranteed that exactly one of the datacenters is configured as the active one.
I think, you can't do it with (native) heartbeat only. You can use pacemaker, he can work with quorums, but... You don't have a quorum. Imagine, that link between data centers fails - every of east and west will think, he is only one survivor and every of them start application, switch mysql to master mode etc. And you'll get really split-brain position.
IMHO, if you need really HA, you need 3th data center, then migrate MySQL to MariaDB with Galera cluster, and start on them your Java app, may be even in active-active-active mode.
The solution I came up with is to keep two versions of /etc/ha.d/haresources.
The "haresources-dark" is used in all servers in the DR datacenter (east). I use a symlink so haresources points to haresources-dark.
The only difference between the two versions of haresources is the mention of Java applications. In the dark version, Java applications are not started.
If/when we ever switch to the DR datacenter, we'll have to update these symlinks manually. But that is acceptable.
This is not mistake-proof. I have to manually set up the symlinks on all my heartbeat-managed servers in the DR datacenter. And there's nothing to enforce that one datacenter is "dark" and the other is "live." This is going to be a manual solution for now.