Currently our system runs entirely inside AWS. We do rolling snapshots of our EBS and often practice running restores.
What keeps me up at night is having all our eggs in one basket. Here are the scenario's:
- Our Amazon zone has some massive event that destroys the data centre
- Someone gains access to our AWS account, terminates our instances and deletes all our snapshots
To mitigate these risks I'm thinking about moving snapshots periodically to another AWS account (with different credentials) in another region.
My question is this, is this an adequate level of precaution or should I be looking to offsite backups that are completely removed from Amazon?
This is risk assessment, not professional systems administration. There's, arguably, a technical component to this decision, but it's fundamentally a business (and dollars and cents) decision.
I think it's wise to plan for both scenarios if it can be handled cost-effectively. The second scenario seems vastly more likely than the first, but they're both plausible.
If it were me, I'd lobby hard for offsite backups that were completely removed from Amazon. A third, perhaps more likely scenario than your first two, might involve the business relationship between your company and Amazon going sour. While there are certainly legal remedies in that situation it would be advantageous for you if you could continue business operations with another hosting provider while things play out with Amazon. To that end, having backups (at minimum) that are accessible without Amazon's involvement seems prudent.
(I'd even argue that it's probably worth doing an assessment of spinning up your entire application on another host. If things did go sour with Amazon having backups is nice, but it would be nicer still if you could continue running your site. That may be pie-in-the-sky depending on how deeply integrated your application is to Amazon's platform, but it's at least worth discussing.)
It looks like your threat model is pretty good.
AWS provides availability zones on EC2 instances (and related services) to help guard against this type of thing a little bit. Putting backups in another region is even better.
It's not so much about Amazon's immunity or non-immunity to damage, but about the concept of backups being separated by physical distance.
The threat model related to someone compromising your account is totally reasonable; people have been held for ransom that way in the past.
Personally I wouldn't move the snapshots. If you are using EC2 servers as anything but ephemeral nodes, you aren't using the full power of AWS. The point of a "cloud architecture" is that the servers can be zapped at any time. But, I would definitely apply this paradigm to actual backups (dumps of data, etc.).
Your alternative to putting backups into a separate account and probably a separate AWS region is much more expensive: an offsite data storage company. Those guys are usually rather more costly, but tend to make explicit statements about how safe your data is, in exchange.