NOTE: My understanding of this question has changed significantly since I first asked it (see Edit 2 below) but I've left the original version intact.
We've put together an offsite backup system (still testing in-house) that does its data transfer via ZFS send/receive. Machines on both ends are FreeBSD 8.2. Overall, the setup works well.
However, there's clearly something I don't understand about ZFS snapshot stream sizes. I've had a hard time finding information on this, so I'm hoping someone with some more experience can enlighten me.
On the source machine, I have a filesystem of about 47GB for which I need to transfer snapshots:
# zfs list -t snapshot -r -s creation stg/serverx
NAME USED AVAIL REFER MOUNTPOINT
(.......)
stg/serverx@20110620 2.88M - 47.1G -
stg/serverx@20110621 2.89M - 47.1G -
stg/serverx@20110622 2.88M - 47.1G -
stg/serverx@20110623 5.44M - 46.6G -
I have the snapshot from 6/22 already on the remote server, so I send it the stream generated by
zfs send -i stg/serverx@20110622 stg/serverx@20110623
This is received on the other end with no trouble; however, the stream generated is over 80 gigabytes—nearly twice the size of the entire source filesystem.
Am I misinterpreting the meaning of the "USED" column generated by zfs list? I'd have expected this snapshot stream to be 5.44M plus a certain amount of overhead. It seems I don't quite understand what constitutes overhead.
Possibly useful information: We backup (via rsync) each server to its own filesystem. This particular one seems to generate the largest streams (relative to filesystem and snapshot size). I suspect it may be related to the fact that it's a mail server, so some of its content is pretty dynamic. However, I'd expect this to show up in the snapshot "used" size too.
Obviously, we can save quite a bit by compressing the stream (it can probably be reduced to 12-20% of its original size). Even then, bandwidth will be our limiting factor, so we'd like to understand what makes these streams so large and whether we can do anything to mitigate it.
EDIT: I had forgotten that we have zfs compression enabled on the source filesystem. So the 47-ish gigabytes used does indeed translate to nearly 80 gigabytes of "real" filesystem data. I suppose that's a partial explanation, but I still don't see why the incremental stream from zfs send would be so large.
EDIT 2:
Further investigation of this backup and a couple of others has led to the conclusion that the large transfers were in fact to be expected (due to some upgrades that had taken place). However, I don't see any indication of that large amount of data in the output from zfs list.
I've been through documentation, and I understand that there are a lot of complexities in computing the space used by a snapshot. The zfs man page says the following in the description of the used property:
When snapshots . . . are created, their space is initially shared between the snapshot and the file system, and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot, and counted in the snapshot's space used.
This makes sense to me. However, I would expect to see a much larger snapshot created at the end of the day the server was upgraded. In fact, it's only a few megabytes. There is no deduplication here (zpool version 15). However, the incremental stream generated by zfs send -i is quite large, and contains all the upgrade information.
Can someone explain this apparent inconsistency? The related question, then, is: How can I get a reasonable estimate of the size of an incremental stream (from, for example, zfs list output)?
I know this is a really old question, but I've seen it a few difference places. There's always been some confusion about the value expressed in zfs list as it pertains to using zfs send|recv. The problem is that the value expressed in the USED column is actually an estimate of the amount of space that will be released if that single snapshot is deleted, bearing in mind that there may be earlier and later snapshots referencing the same data blocks.
Example:
In order find out how much data will need to be transferred to reconstitute a snapshot via zfs send|recv, you'll need to use the dry-run feature (-n) for these values. Taking the above-listed snapshots try:
Yikes! That's a whole heck of a lot more than the USED values. However, if you don't need all of the intermediary snapshots at the destination, you can use the consolidate option (-i rather than -I), which will calculate the necessary differential between any two snapshots even if there others in between.
So that's isolating the various blocks that were rewritten between snapshots, so we only take their final state.
But that's not the whole story! zfs send is based on extracting the logical data from the source, so that if you have compression activated on the source filesystem, the estimates are based on the uncompressed data that will need to be sent. For example, taking one incremental snapshot and writing it to disk you get something close to the estimated value from the dry-run command:
But if you pass it through gzip, we see that the data is significantly compressed:
Side note - this is based on the OpenZFS on Linux, version : - ZFS: Loaded module v0.6.5.6-0ubuntu16
You will find some references to optimisations that can be applied to the send stream (-D deduplicated stream, -e more compact), but with this version I haven't observed any impact on the size of the streams generated with my datasets.
What type of email system and what type of "store" technology? If the mail store is already compressed in any way, then each incremental may actually be a full as its compressed data stream may be dynamically changing due to its compression.
Also is dedup in play across either system? It sounds like there may be a remote chance that it might be on the source system. That might account for the size difference.