I am looking to setup a failover in the following configuration -
Failover Server (Frontend)
Primary Server - Secondary Server (used if primary server fails) (Backend)
So traffic goes to the failover and then is redirected to the primary server if Primary Server is up, otherwise it will be routed to the Secondary Server.
Firstly is this the best way to achieve what I am trying and secondly, what if the frontend fails?
Basically you have both forward and rear-facing VIPs. As an example;
Two Web Servers x.x.x.1 & x.x.x.2 have a client-facing VIP of x.x.x.3, this last one is the one that the DNS records point to and is managed either by the servers themselves or another load-balancer depending on your OS/implementation.
They also have a rear/internal-facing VIP of say y.y.y.1 that load-balances to their internal IPs y.y.y.1 & y.y.y.2. The web servers talk 'down' to the internal tier VIP (z.z.z.3) via these last two interfaces but are spoken 'up' to by the internal servers on this y.y.y.3 VIP.
The internal servers might be z.z.z.1 & z.z.z.2 with an upward/web-facing VIP of z.z.z.3 and although they each talk up to the web servers via the y.y.y.3 VIP and are spoken to by these servers on the z.z.z.3 VIP.
Of course you can then add a fourth/fifth layer down below this to allow your interal servers to talk to DB servers which may have their own VIPs.
Either way you're using load-balancing/VIPs throughout to allow for failure scenarios as you discussed to be dealt with.
When we do it, the client can have any number of servers. If all of those fail, their records point at our "courtesy server". All it says is "example.com is currently not available please check back in a few minutes".
It's not pretty, but more reassuring to the normal user than a perpetually spinning globe.