Is it safe to delete oplog local.* files from a MongoDB snapshot ? Does the fsyncLock() commit oplog entries in DB files ?
I currently do a fsyncLock() + LVM snapshot of Mongo data. Then i mount the snapshot, and remove the oplog local.* files before archiving the files to S3 or FTP (with Duplicity). I do that to reduce the size of archives to transfer to S3.
When i restore on a node on an empty replicaset, oplog files are recreated, and other nodes full resync from this new primary. But i am not sure if there could be missing data in the restore, due to it being in the previously deleted oplog and not written in DB files.
Using snapshot functionality for mongo replication back up is not good idea while you run it. First of all you need to get the snapshot of "primary" in your replica set. but the problem is your clients can connect with a wrong write concern policy. like not asking for it or not for majority that leaves uncommitted data in your primary's oplog. even if you keep oplog or all files in local db, you may have problems when you restore (got that once, my primary was looking a rollback with no reason).
my suggestion (i assume you can not stop mongod) is using mongodump, yes it sounds stupid when you can have a snapshot. but the real point of using backups is able to restore them.
if you can't use mongodump or don't want to. Try MMS's cloud backup feature. which can restore whole replica set easily.
Edit: Another option is running a delayed replica member. here is documentation. which covers most of the backup needs. just determine the delay time. by this way you don't eat up your server's io with running backup/archive tools.