I have a website that we load-balance across a few machines. The load-balancer (a Brocade ServerIron ADX) is on the local network. I know it has the capability to configure a "backup" ip address to use as the "real", but it would need to be on a local network. How can I provide a "fail whale" to my users and a status update message when my entire environment may be offline due to a FW, LB, or multiple-server failure? Since DNS doesn't have the ability to provide a weighted A-Record backup (like SRV records), what are my options?
Ideally, the solution should be able to step in automatically as soon as my site becomes unreachable and inform my users that our response teams on working on the problem.
Twitter's Fail Whale is more complicated then it seems. The Twitter applications stacks (Stacks-- there are many layers in the Twitter infrastructure) are known to be quite complicated. Twitter has thousands of machines, years worth of code written in various languages, glue in a dozen different flavors, and hundreds (if not thousands) of places where the application can break. Your requirements (two servers, and a failure page) is much simpler.
I just happen to be looking at a similar feature-- I want to use a backup webserver if all of the primary webservers fail. Note that this will only help in some failure scenarios, and won't help if the public network to the loadbalancer has problems.
ServerIron ADX Server Load Balancing Guide, Chapter 2: section "Primary and backup servers" says:
You get a web designer to produce a document which looks like you want it to look.
Then you configure the load balancer to serve this document for HTTP 500, 502, 503 and 504 errors. Exactly how you do that varies; check your documentation.
Some load balancers are just "dumb" TCP balancers which forward TCP connections and do nothing at layer 7. Others can act as full reverse proxies (e.g. nginx) and these are capable of doing what you need.
From a quick scan of the Brocade ServerIron ADX documentation, it doesn't seem capable of serving error documents on HTTP requests. While it's definitely smarter than your typical "dumb" TCP forwarder, it's probably not going to do what you want here.
A reverse proxy such as nginx would be capable of this, though if you set that up, you may as well just replace the load balancer (since nginx can do HTTP/HTTPS load balancing as well).
Twitter's 'fail whale' does not indicate any of these catastrophic failures, and they would not be trivial to create at all. Best I can think of is having a low DNS TTL for your domain, an extra internet connection with separate IP space that serves just the fail whale and some kind of monitoring tool that updates your A records upon failure.
Now unless you plan to have catastrophic failures very frequently, this would be overkill. And if you do plan to have them frequently, you're probably doing it wrong :)
CDN's such as CloudFront or Akamai can substitute a fail-whale for error responses, or you can have a lightweight proxy layer on-premise that does the same thing. The local solution won't help you if your network connection dies, only a CDN or remotely hosted DNS server+healthchecker can handle that.