Home / server / Questions / 938780
Accepted
IMTheNachoMan
Asked: 2018-11-06 18:58:02 +0800 CST

ZFS pool size discrepancy after rsync-ing

  • 772

I am trying to rsync data one of the volumes in a ZFS pool to an exfat formatted drive. The volume is only 1.3 TB but the rsync got up to syncing 3+ TB worth of data. I killed the rsync so I could figure out what is wrong.

The ZFS pool:

ubuntu@ubuntu:~$ zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
backup  7.25T  3.59T  3.66T         -     0%    49%  1.00x  ONLINE  -

The volumes and their mountpoints:

ubuntu@ubuntu:~$ zfs list
NAME                                                      USED  AVAIL  REFER  MOUNTPOINT
backup                                                   2.61T  2.49T  35.4M  /backup
backup/.system                                           1.97M  2.49T   140K  legacy
backup/.system/configs-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx   593K  2.49T   593K  legacy
backup/.system/cores                                      692K  2.49T   692K  legacy
backup/.system/rrd-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       128K  2.49T   128K  legacy
backup/.system/samba4                                     337K  2.49T   337K  legacy
backup/.system/syslog-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx    128K  2.49T   128K  legacy
backup/vol1                                              1.26T  2.49T  1.26T  /backup/vol1
backup/vol2                                               128K  2.49T   128K  /backup/vol2
backup/vol3                                              1.78G  2.49T  1.78G  /backup/vol3
backup/vol4                                              1.34T  2.49T  1.34T  /backup/vol4
backup/vol5                                              4.51G  2.49T  4.51G  /backup/vol5

The size on disk of /backup/vol1:

ubuntu@ubuntu:~$ du -sh /backup/vol1
1.3T    /backup/vol1

How much data was copied after doing an rsync for a couple days:

ubuntu@ubuntu:~$ rsync -avzh --progress --no-o --no-g /backup/vol1 /media/ubuntu/external_drive/freenas/
...
...
ubuntu@ubuntu:~$ du -sh /media/ubuntu/external_drive/freenas/vol1
3.2T    /media/ubuntu/external_drive/freenas/vol1

I killed it after seeing how much was transferred.

rsync

1 Answers

  1. Best Answer

    Just to expand on my comment above, this will happen under two circumstances that I can think of:

    • The files / volumes you're sending are using block saving techniques (compression, dedup, or snapshots/clones). ZFS has saved you some space internally, so the ZFS property used is small, but rsync still has to read all the logical (uncompressed, undeduped, un-snapshot-sharing) data because it has no idea about those features. You can figure out how much logical data there is by running zfs list -o logicalused (or logicalreferenced if you have snapshots).
    • The files / volumes you're trying to send are sparse. From your comment above, I believe this is the case you're in. rsync is sending all the unwritten blocks of your volume, even though they're all zeroed out (because they've never been written to). Try using the --sparse argument to rsync, as described in this answer, to work around this behavior, and then using --in-place for subsequent syncs to avoid re-syncing the entire file instead of just the parts that have changed.
    • 0