I started to get "not enough free space" errors when trying to copy files, and at some point I had trouble booting and did a ZFS rollback. I still get space errors despite (according to df and du below) <60% drive space being used.
The answer below points to zfs snapshots taking up too much space, and it's a helpful start. But sudo du /.zfs/snapshot/ -h --max-depth=0 returns only 60G. Are there 140G of snapshots elsewhere? (or what else could it be?)
Also conspicuously zfs list -t snapshot doesn't show the (biggest and I assume most defunct?) dataset rpool/USERDATA/thomas_fv2wpz which is shown in zfs list output below to be 278G.
Here's a table of contents for outputs:
uname -azfs listzfs list -o space rpool/USERDATA- Should I omit (some/parts of) the following long outputs? this my first time asking a question.
df -hdu -h --max-depth=1lsblk -f
$ uname -a
Linux thomas-Latitude-E6530 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
bpool 272M 1.48G 96K /boot
bpool/BOOT 270M 1.48G 96K none
bpool/BOOT/ubuntu_nhdx6z 270M 1.48G 180M /boot
bpool/BOOT/ubuntu_smztg7 8K 1.48G 180M /boot
rpool 441G 4.89G 96K /
rpool/ROOT 8.97G 4.89G 96K none
rpool/ROOT/ubuntu_nhdx6z 8.51G 4.89G 3.69G /
rpool/ROOT/ubuntu_nhdx6z/srv 264K 4.89G 96K /srv
rpool/ROOT/ubuntu_nhdx6z/usr 872K 4.89G 96K /usr
rpool/ROOT/ubuntu_nhdx6z/usr/local 776K 4.89G 144K /usr/local
rpool/ROOT/ubuntu_nhdx6z/var 3.24G 4.89G 96K /var
rpool/ROOT/ubuntu_nhdx6z/var/games 152K 4.89G 96K /var/games
rpool/ROOT/ubuntu_nhdx6z/var/lib 3.10G 4.89G 2.74G /var/lib
rpool/ROOT/ubuntu_nhdx6z/var/lib/AccountsService 688K 4.89G 104K /var/lib/AccountsService
rpool/ROOT/ubuntu_nhdx6z/var/lib/NetworkManager 1.76M 4.89G 164K /var/lib/NetworkManager
rpool/ROOT/ubuntu_nhdx6z/var/lib/apt 70.1M 4.89G 53.6M /var/lib/apt
rpool/ROOT/ubuntu_nhdx6z/var/lib/dpkg 97.8M 4.89G 39.4M /var/lib/dpkg
rpool/ROOT/ubuntu_nhdx6z/var/log 138M 4.89G 36.4M /var/log
rpool/ROOT/ubuntu_nhdx6z/var/mail 152K 4.89G 96K /var/mail
rpool/ROOT/ubuntu_nhdx6z/var/snap 1.31M 4.89G 240K /var/snap
rpool/ROOT/ubuntu_nhdx6z/var/spool 848K 4.89G 120K /var/spool
rpool/ROOT/ubuntu_nhdx6z/var/www 152K 4.89G 96K /var/www
rpool/ROOT/ubuntu_smztg7 471M 4.89G 3.18G /
rpool/ROOT/ubuntu_smztg7/srv 56K 4.89G 96K /srv
rpool/ROOT/ubuntu_smztg7/usr 144K 4.89G 96K /usr
rpool/ROOT/ubuntu_smztg7/usr/local 144K 4.89G 144K /usr/local
rpool/ROOT/ubuntu_smztg7/var 203M 4.89G 96K /var
rpool/ROOT/ubuntu_smztg7/var/games 112K 4.89G 96K /var/games
rpool/ROOT/ubuntu_smztg7/var/lib 177M 4.89G 2.65G /var/lib
rpool/ROOT/ubuntu_smztg7/var/lib/AccountsService 144K 4.89G 104K /var/lib/AccountsService
rpool/ROOT/ubuntu_smztg7/var/lib/NetworkManager 320K 4.89G 148K /var/lib/NetworkManager
rpool/ROOT/ubuntu_smztg7/var/lib/apt 4.08M 4.89G 53.3M /var/lib/apt
rpool/ROOT/ubuntu_smztg7/var/lib/dpkg 14.0M 4.89G 39.4M /var/lib/dpkg
rpool/ROOT/ubuntu_smztg7/var/log 24.9M 4.89G 15.0M /var/log
rpool/ROOT/ubuntu_smztg7/var/mail 112K 4.89G 96K /var/mail
rpool/ROOT/ubuntu_smztg7/var/snap 212K 4.89G 188K /var/snap
rpool/ROOT/ubuntu_smztg7/var/spool 160K 4.89G 120K /var/spool
rpool/ROOT/ubuntu_smztg7/var/www 112K 4.89G 96K /var/www
rpool/USERDATA 432G 4.89G 96K /
rpool/USERDATA/helper_zthl10 180K 4.89G 124K /home/helper
rpool/USERDATA/root_fv2wpz 316K 4.89G 1.12M /root
rpool/USERDATA/root_mhe8ya 1.78M 4.89G 992K /root
rpool/USERDATA/thomas_fv2wpz 971M 4.89G 278G /home/thomas
rpool/USERDATA/thomas_mhe8ya 431G 4.89G 217G /home/thomas
$ zfs list -o space rpool/USERDATA
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
rpool/USERDATA 4.88G 432G 0B 96K 0B 432G
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 5.8G 0 5.8G 0% /dev
tmpfs 1.2G 2.1M 1.2G 1% /run
rpool/ROOT/ubuntu_nhdx6z 8.8G 3.7G 5.1G 42% /
tmpfs 5.8G 253M 5.6G 5% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
rpool/USERDATA/helper_zthl10 5.1G 128K 5.1G 1% /home/helper
bpool/BOOT/ubuntu_nhdx6z 1.7G 180M 1.5G 11% /boot
rpool/USERDATA/thomas_fv2wpz 284G 279G 5.1G 99% /home/thomas
rpool/USERDATA/root_fv2wpz 5.1G 1.0M 5.1G 1% /root
rpool/ROOT/ubuntu_nhdx6z/srv 5.1G 128K 5.1G 1% /srv
rpool/ROOT/ubuntu_nhdx6z/var/games 5.1G 128K 5.1G 1% /var/games
rpool/ROOT/ubuntu_nhdx6z/var/log 5.2G 29M 5.1G 1% /var/log
rpool/ROOT/ubuntu_nhdx6z/var/lib 7.9G 2.8G 5.1G 35% /var/lib
rpool/ROOT/ubuntu_nhdx6z/usr/local 5.1G 256K 5.1G 1% /usr/local
rpool/ROOT/ubuntu_nhdx6z/var/mail 5.1G 128K 5.1G 1% /var/mail
rpool/ROOT/ubuntu_nhdx6z/var/www 5.1G 128K 5.1G 1% /var/www
/dev/sda1 511M 7.1M 504M 2% /boot/efi
rpool/ROOT/ubuntu_nhdx6z/var/snap 5.1G 256K 5.1G 1% /var/snap
rpool/ROOT/ubuntu_nhdx6z/var/spool 5.1G 128K 5.1G 1% /var/spool
rpool/ROOT/ubuntu_nhdx6z/var/lib/AccountsService 5.1G 128K 5.1G 1% /var/lib/AccountsService
/dev/loop2 62M 62M 0 100% /snap/caprine/36
/dev/loop3 55M 55M 0 100% /snap/core18/1754
/dev/loop4 176M 176M 0 100% /snap/gimp/273
/dev/loop0 9.2M 9.2M 0 100% /snap/canonical-livepatch/95
/dev/loop5 241M 241M 0 100% /snap/gnome-3-34-1804/24
rpool/ROOT/ubuntu_nhdx6z/var/lib/dpkg 5.2G 40M 5.1G 1% /var/lib/dpkg
rpool/ROOT/ubuntu_nhdx6z/var/lib/apt 5.2G 54M 5.1G 2% /var/lib/apt
rpool/ROOT/ubuntu_nhdx6z/var/lib/NetworkManager 5.1G 256K 5.1G 1% /var/lib/NetworkManager
/dev/loop7 174M 174M 0 100% /snap/gimp/252
/dev/loop6 94M 94M 0 100% /snap/core/9066
/dev/loop1 55M 55M 0 100% /snap/core18/1705
/dev/loop8 164M 164M 0 100% /snap/spotify/41
/dev/loop9 256K 256K 0 100% /snap/gtk2-common-themes/9
/dev/loop10 143M 143M 0 100% /snap/slack/23
/dev/loop11 256M 256M 0 100% /snap/gnome-3-34-1804/33
/dev/loop13 63M 63M 0 100% /snap/gtk-common-themes/1506
/dev/loop12 28M 28M 0 100% /snap/snapd/7264
/dev/loop14 50M 50M 0 100% /snap/snap-store/454
/dev/loop15 142M 142M 0 100% /snap/zoom-client/83
/dev/loop16 161M 161M 0 100% /snap/gnome-3-28-1804/116
/dev/loop17 87M 87M 0 100% /snap/shotcut/64
/dev/loop18 291M 291M 0 100% /snap/vlc/1620
/dev/loop19 140M 140M 0 100% /snap/slack/24
/dev/loop20 50M 50M 0 100% /snap/snap-store/433
/dev/loop21 145M 145M 0 100% /snap/zoom-client/84
tmpfs 1.2G 52K 1.2G 1% /run/user/1000
/dev/loop22 64M 64M 0 100% /snap/caprine/37
$ du -h --max-depth=1
25K ./Music
1.0K ./.ssh
1.0K ./Templates
36K ./.gnome
30M ./.thunderbird
52M ./Documents
368M ./snap
8.4M ./.mozilla
203G ./Pictures
302M ./.config
1021M ./.cache
1.0K ./Public
9.5G ./Downloads
101M ./.local
7.0K ./Steam
41G ./Videos
28K ./.gnupg
7.6G ./Desktop
16G ./.steam
17K ./.pki
11K ./.psensor
279G .
$ lsblk -f
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
loop0 squashfs 0 100% /snap/canonical-livepatch/95
loop1 squashfs 0 100% /snap/core18/1705
loop2 squashfs 0 100% /snap/caprine/36
loop3 squashfs 0 100% /snap/core18/1754
loop4 squashfs 0 100% /snap/gimp/273
loop5 squashfs 0 100% /snap/gnome-3-34-1804/24
loop6 squashfs 0 100% /snap/core/9066
loop7 squashfs 0 100% /snap/gimp/252
loop8 squashfs 0 100% /snap/spotify/41
loop9 squashfs 0 100% /snap/gtk2-common-themes/9
loop10 squashfs 0 100% /snap/slack/23
loop11 squashfs 0 100% /snap/gnome-3-34-1804/33
loop12 squashfs 0 100% /snap/snapd/7264
loop13 squashfs 0 100% /snap/gtk-common-themes/1506
loop14 squashfs 0 100% /snap/snap-store/454
loop15 squashfs 0 100% /snap/zoom-client/83
loop16 squashfs 0 100% /snap/gnome-3-28-1804/116
loop17 squashfs 0 100% /snap/shotcut/64
loop18 squashfs 0 100% /snap/vlc/1620
loop19 squashfs 0 100% /snap/slack/24
loop20 squashfs 0 100% /snap/snap-store/433
loop21 squashfs 0 100% /snap/zoom-client/84
loop22 squashfs 0 100% /snap/caprine/37
sda
├─sda1 vfat 105C-066C 503.9M 1% /boot/efi
├─sda2
├─sda5 swap ca95a35a-589b-4127-91d0-ba74f05c7011 [SWAP]
├─sda6 zfs_member bpool 10888252588400514123
└─sda7 zfs_member rpool 8231086401312639965
sr0
Ubuntu 20.04 introduced experimental ZFS support with
zsys(ZFS SYStem integration) tool. It creates snapshots on mostaptoperations for that you will be able to rollback a previous state if something fails.Downside is of course that these will also eat up your available space.
Use
zfs list -t snapshotto list the snapshots:and
zfs destroyto delete a snapshot, e.g.:Read also the official tutorial.
There is also a similar question with not many answers yet on how to disable automatic snapshot creation and another one asking for some official documentation (I couldn't find anything either).
You could also use this
forloop I wrote to just wipe out all of a specific pool. I wiped outrpoolandbpoolas I take backups of my PC with something else daily anyway. Just be sure to change frombpooltorpoolif you only want one or the other.