We have two mailservers ("dubone" and "dubdeuce"), one does all the work while the other sits idle. We want to have a setup where if dubone is busy or down, dubdeuce can accept/reject/filter emails just as dubone would.
Then, once dubone is available again, dubdeuce would push all of the emails it has stored onto dubone, so that the emails are all in one place.
We are using Exim4 and Dovecot backed by MySQL, and accessible with RoundCube. Both machines run Ubuntu Linux.
How can I sync the databases between both machines so that dubdeuce is always current (which it needs to be to do its job correctly), and then how would I instruct dubdeuce to push all of its accumulated messages on to dubone?
There's three components to what you're asking here: Database replication, mail redundancy, and user access.
Database replication is easy, and is well covered by the MySQL documentation.
Mail replication is generally just as easy as making sure your MX records are set correctly. If dubdeuce is not the primary MX, it'll forward mail to the primary MX once the primary is available again.
However, you haven't explicitly stated it, but your inclusion of roundcube implies that you want the second system to be fully operational as far as a MUA goes - you want users to be able to read email. If that's not the case - you're happy for roundcube to be offline until dubone comes back - then the above components will do the job for you.
Once more: if all you want is a simple system that will store-and-forward email until the primary comes back, all you need is database replication for mysql and to make use of a secondary MX. This system should be live at all times, not just spun up at need.
If you want a fully redundant system, in which roundcube is always available, then what you want is a shared mailspool between the two systems. There's no real way to configure dubduece as a primary MX and give users the ability to read mail from it, and have it sensibly forward email to dubone.
So, you need a shared mailspool. This could be a third system acting as fileserver, serving the mailspool to the two primary hosts, however there's a longstanding recommendation against using NFS for a mailspool, due to locking issues.
It could be done using DRBD between the two nodes, in active/backup mode - when one node fails, you use a heartbeat to switch the other node to active. When the first node comes back online, you'll need a heartbeat process to switch everything back as well. You'll still need to work out how you replicate your database - you might want a multimaster setup now.
Finally, you could do the same thing with DRBD, but use a cluster-aware filesystem on it, and have both nodes being active at all times.That's a bit more complicated though. You could DRBD the entire mail system between nodes too. And there's plenty of ways to scale up - more advanced solutions involve a SAN and a VM stack like Citrix Xenserver or VMware.
For my money, I'd stick with an active/passive DRBD mailspool, either multimaster mysql, or DRBD active/passive backed mysql, and use heartbeat to move enable active services on failover. An alternative to this is putting your whole mailserver into a VM using Xen or KVM or whatever you like, backing the VM onto a DRBD system, and having heartbeat failover the DRBD and start up the VM on the second node in the case of a failure. In this example, you've only actually got "one" mailserver, it just floats between your nodes. The downside is that you have to wait for it to boot on failover, which may take a while.
As a side note, whichever way you do things, make sure that your spam / virus / malware / etc scanning is configured identically on both systems.