How can I guard against an administrator deleting both a server and its cloud backup at the same time?

In some envs, you simply split the job into two teams - one team runs/guards the servers, the other runs/guards the backups.

the other part is, imo even more, important: if the backup is writeable, especially from the system that is being backed up, it is NOT a good backup.

systemic issue in many tools like borg. it is best if you can (in a cloud world) push the backups forward to AWS glacier, while storing them into S3 with a role that can only create them. there's schedules for deletion and noone needs to be able to "quickly" do that.

also, don't forget that there's books about this stuff.

Here's how to do it:

1. On Backblaze B2, create an application key that cannot delete files:

   b2 create-key --bucket MyBucket MyKeyName listBuckets,listFiles,readFiles,writeFiles
   

2. Setup the backup so that it uses that key and does not attempt to delete old backup files. For example, in duplicity, do not use remove-older-than, remove-all-but-n-full, or remove-all-inc-of-but-n-full; in duply, do not use purge or purgeIncr.

3. To delete old backups, set custom lifecycle settings on the bucket; for example, set lines beginning with duplicity-full to be hidden after 360 days and deleted after another 360; and so on for files beginning with duplicity-inc and duplicity-new.

Update:

B2 does not actually provide any functionality to "delete a file". Each time you replace the same file, it keeps its history, so a file can have "versions"—the most recent one is normally the one you need. What B2 provides is functionality for "hiding" a file. When you hide a file, you are actually recording in the file's history that the file was "deleted", you kind of add a new file version which is the hidden, or deleted, file.

Except for that, B2 also provides functionality to actually delete file versions. A user who does not have the deleteFiles permission actually does have permission to hide files, but not to delete file versions.

It seems to me that duplicity's remove-... functionality should be implemented by hiding files. (It would then be up to the bucket's configuration to actually delete these hidden file versions after some amount of time.) However duplicity's B2's backend doesn't do this; what it does is to actually delete file versions.

On most public clouds, ssh access to the server does not mean you can delete the server. There is a difference between administering on the server and in the cloud account. So, you can separate these. You can also split out permissions quite a bit, especially on Amazon AWS. You can give someone the permission to reboot the server, but not delete it, for example. And AWS has the delete protect option on VMs.

Best practice, these days, is to use configuration management like Chef/Puppet/Salt/Ansible to do any changes on your servers and not actually log into the servers.

In addition, for AWS, you can snapshot the servers as often as you want and this is the most efficient way to back them up.