Short and sweet, I don't suppose you'll need more detail than this:
We host our website on an in-house webserver.
A catastrophe has and will happen again where communication from the web into/out of our building ceases.
When this happens, we'd like our users to at least see some kind of "down for maintenance" or other message (twitter's Fail Whale?) when trying to access our site, rather than the current "server not found" type messages that they currently get. (A static server, temporary redirects for all requests).
[Being able to fail-over to a fully functional website would be awesome, but is not required, and probably not an option].
Obviously there has to be some kind of off-site failover that could take over when our local servers are down.
How is such a solution implemented? Other suggestions?
EDIT: If I'm not using (or leaving out) some correct terminology for this, let me know!
An external reverse proxy could be configured to redirect users to another server when all fails. This would take care of DNS propagation as well.
Who hosts your DNS? DYNdns.org offers a service where if your IP becomes non-responsive it will display an offline message or redirect to another url. I believe it's called webhop. This is the only one I'm familiar with but if your DNS is hosted that is probably an option. If you host your own I'm not certain.
IPVS (IP Virtual Server) implements transport-layer load balancing inside the Linux kernel, so called Layer-4 switching. IPVS running on a host acts as a load balancer at the front of a cluster of real servers, it can direct requests for TCP/UDP based services to the real servers, and makes services of the real servers to appear as a virtual service on a single IP address.
So you could se an external ipvs+keepalived setup for failover, if both web-services goes down you can specify a "sorry server" in keepalived.conf. You need to use LVS-tun (ipip tunneling) if the hosts are on different subnets.
(source: ccidnet.com)
There is a service called Simple Failover that may do what you want, though there are still issues with DNS caching since it's a DNS-based solution.
I posted a longer answer with more technical details to a similar question in the past which explains the mechanics behind it.