Home / server / Questions / 1029032
In Process
cvlo
Asked: 2020-08-06 14:49:43 +0800 CST

Auto rebooting down servers

  • 772

Currently have a web server VPS in Germany with Hetzner. They are offering free load balancer service. Thinking about cloning my VPS in their Finland site (meaning adding another VPS with cloned setup/contents) for daily backups that can also increase my site's availability through the load balancer.

Is this even possible with two VPS, or would I need an extra one for storage/database? I'm basically trying to accomplish a RAID-1 structure but with two VPS instead of two hard drives.

What's the best way to set this up on Debian 10 in a way that whenever one of the servers is down, the other one can send the order to automatically reboot?

backup load-balancing high-availability redundancy apache2

1 Answers

  1. There's two parts. First is to detect whether the server is down; second is to send the reboot command.

    For detection, I'd suggest using a service that monitors from multiple sites, such as HetrixTools (there's many others too). If you simply monitor from the other VPS, there's a risk of a "false positive" where the other VPS is considered "down" due to some network issue between Hetzner datacenters but accessible from most locations, leading to an unnecessary reboot.

    In order to send the reboot command, use Hetzner's API (look under "Soft reboot a server"). Most monitoring services have a "webhook" notification which you can use to call the API. If not (or if you don't want to disclose your API key to the monitoring service) then you can point the webhook at your own web server and call the API from there.

    More generally, rebooting the server soon as it is detected down is not necessarily the best idea. You don't know the reason (is Hetzner about to perform scheduled maintenance?) and an uncontrolled reboot can lead to more rather than less downtime if the problem is transient. A down notification also typically happens as soon as the server goes down and never again, which is potentially the worst time to reboot. There probably needs to be a bit more sophisticated algorithm, such as waiting X minutes and retrying after at least Y minutes rather than just blindly rebooting on a down detection. But this is beyond the scope of your original question.

    • 0