I'm a small company on not much budget providing websites and databases for charity and not-for-profit clients.
I have a few Debian Linux VPS servers and ensure I have daily backups to a different VPS than the one the service is hosted on.
Recently one of my hosting companies told me two drives failed simultaneously and so that data was lost forever. Stuff happens, they said sorry, what else could they do? But it made me wonder about cost-effective ways to basically get a VPS up again in the event of a hardware or other host-related failure.
Currently I would have to
- Spin up a new VPS
- Get the last day's backup (which includes databases, web root and website-specific config) over onto the VPS, and configure it like the last one etc.
- Update DNS and wait for it to propagate.
It would probably take a day or so achieve this, with the DNS propagation being a big unknown, although I have the TTL set quite low (hour or so).
Some hosts provide snapshots which can be used to replicate a set up to a new VPS, but there's still the IP and this doesn't help in the case that the host company cancels/suspends an account outright (I've been reading about this behaviour from certain hosting providers and it's scared me! I'm not doing anything spammy/dodgy and keep a close eye on security, but I realise that they literally have the power to do this and I'm quite risk averse).
Is this, combined with choosing reputable hosts, the best I can do without going for an incredibly expensive solution?
For me, choosing reputable hosts and doing regular backups - both of which you seem to be doing already - is about as well as you can do without starting to think about business continuity planning, high-availability setups, SLAs, and so on.
I tell people that you get 99% uptime for free (ie, without spending anything extra on high availability). That's about three and half days downtime a year. Every extra 9 on that uptime increases the cost by somewhere between three and ten times.
If people aren't ready to pay that kind of money, it is in my opinion a mistake to mislead them into thinking they can get any extra protection of any significance.
Small businesses with small budgets, especially nonprofits, typically are not going to be able to afford high availability. The question is, if you have virtually no budget, as is commonly the case in situations like this, what is your restore strategy?
I do have some clients like this, and this is what I do:
First, for some of them I have an incremental backup and full database dump every six hours. One client was already using CrashPlan Pro so I just used that. Whatever you do, you need to make sure you have a restorable backup.
I have a simple ansible playbook I put together in about an hour (not having previously worked with ansible) that installs nginx, php-fpm and MariaDB and prepares them to host a web site or sites. Running this playbook results in a server (or servers) that are ready to host a typical web application, and I can simply restore the nginx virtual host, application files and database to it.
The result of this is that I can bring up such a web site from backup in just a few minutes, as opposed to the manual way which could take an hour or more.
The complexity of the implementation depends on the application stack, but ideally you'd want to setup a "hot standby" (at a different provider), with data being replicated in real-time (or as close to real time) as possible.
Making the business case for having 2 "live" servers is as simple as comparing the potential loss of revenue during a "recovery from images" period to the expense of another server.
Remember that uptime is not the same as data integrity. You can have 99.99% uptime and have lost all of your data twice in a year as long as the server was restarted "soon enough". Most of the VPS providers are guaranteeing that your server is running, NOT that your data is safe. Your data is Your problem :(.
What you're looking for is something that will store your backups on a separate server and (IMHO) not even in the same provider. Depending on the data size that you're talking about, a portable hard drive could be used as a third line of offline defence. Backup your data as you have been doing and then regularly copy that (or just the changes if possible) to the portable hard drive or even a local computer. There are also reasonably cheap options like Backblaze for backup solutions, but the price will depend on the amount of data you're talking about. If you can do incremental backups it will be much cheaper than full backups, but incremental backups can be very difficult depending on where the data is stored (flat files = easy, database = not so easy).
The answer totally depends on you architecture and requirements. Some time ago 3 discs failed on a server of mine, taking down 20+ vm's when a Raid 6 failed.
I wrote about it at
https://www.linkedin.com/pulse/20140827173324-2064263-how-i-nearly-lost-my-business-to-3-hard-discs
But: Because this is critical, we had backups - daily for non-important stuff, 15 minutes for databases and emails. Heck, now I added a server that gets replicated to another machine every 30 seconds.
You say nothing about the stack, nothing about any budget - so the best and only advice here is to go to some cloud provider and start using their backup mechanisms. But start defining what you actually - need.
Also - the budget for this backup should be in your pricing. It needs to be paid. And whatever infrastructure you need.... you need it. It is not "ridiculous expensive" then.