We have a small network of servers for which DNS service is critical. However, it seems to be a pain to set up redundant DNS service.
What we currently have is two caching/forward resolving servers set up running Ubuntu with Unbound. With the standard resolv.conf set-up, it seems the best we can do is configure a very short timeout.
I'm finding very little help with Google. The solution that seems to be most common is to create a virtual IP or set up heartbeat. But I'm not sure that'd work in our case, because the forwarding servers are in separate subnets and physical locations.
Regardless, I'm wondering:
Wouldn't it make sense to have stub resolvers on each host fire queries in parallel? Nobody seems to be doing this, perhaps for a good reason I'm missing.
Are there good solutions for this besides heartbeat?
Install two LVS balancer(with VRRP) and Keepalived.
Can anycast DNS work for you? I can't really tell from the info in your question.
In any case anycast DNS has been discussed on serverfault before so if you search for it, you should be able to find more info, either on serverfault or the interwebs.