We are currently doing a complete remote backup of our VM's once a week. This takes the whole weekend, and I'd like to speed up the process some if possible.
We are backing up the VM's using Dedup, and the backup size is 300GB using a 128KB block size. Our small office connection can do 10mbit upstream at its best, and this would roughly translate to 3 days for 300GB.
Do you think 128kb is too large a block size for this application? Should it be lowered at the cost of system overhead and the benefit of smaller backup volume?
Can we do differential backups via rsync with Dedup, eliminating the need for complete backups?
Any and all advise/suggestions are welcome.
Thanks
I assume you expect biggest gain from dedup to come from the OS part of VM images. In that case if your VMs aren't clones of each other, then I'd say that 128 KiB is too big block for optimal dedup. If your bottleneck is the network and more efficient deduplication would be helpful, I'd go way down. If you are deduping disk images, then optimum size would be the minimum VM OS-level unit of allocation. In Linux that's a 4KB block (by default) in ext3 and ext4 filesystems. With bigger block sizes pay attention to partitioning, you may have identical systems shifted by half the dedup-block-size because of different virtual disk layouts.
It's hard to give better answer with a rather vague question.
VMware ESX4+ has changed block tracking, which allows backup software to determine which blocks to back up, rather than the entire disk. The backup software can then roll the changes in to the first full backup, essentially eliminating the need for doing a full every week.