I started to get "not enough free space" errors when trying to copy files, and at some point I had trouble booting and did a ZFS rollback. I still get space errors despite (according to df
and du
below) <60% drive space being used.
The answer below points to zfs snapshots taking up too much space, and it's a helpful start. But sudo du /.zfs/snapshot/ -h --max-depth=0
returns only 60G
. Are there 140G of snapshots elsewhere? (or what else could it be?)
Also conspicuously zfs list -t snapshot
doesn't show the (biggest and I assume most defunct?) dataset rpool/USERDATA/thomas_fv2wpz
which is shown in zfs list
output below to be 278G
.
Here's a table of contents for outputs:
uname -a
zfs list
zfs list -o space rpool/USERDATA
- Should I omit (some/parts of) the following long outputs? this my first time asking a question.
df -h
du -h --max-depth=1
lsblk -f
$ uname -a
Linux thomas-Latitude-E6530 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
bpool 272M 1.48G 96K /boot
bpool/BOOT 270M 1.48G 96K none
bpool/BOOT/ubuntu_nhdx6z 270M 1.48G 180M /boot
bpool/BOOT/ubuntu_smztg7 8K 1.48G 180M /boot
rpool 441G 4.89G 96K /
rpool/ROOT 8.97G 4.89G 96K none
rpool/ROOT/ubuntu_nhdx6z 8.51G 4.89G 3.69G /
rpool/ROOT/ubuntu_nhdx6z/srv 264K 4.89G 96K /srv
rpool/ROOT/ubuntu_nhdx6z/usr 872K 4.89G 96K /usr
rpool/ROOT/ubuntu_nhdx6z/usr/local 776K 4.89G 144K /usr/local
rpool/ROOT/ubuntu_nhdx6z/var 3.24G 4.89G 96K /var
rpool/ROOT/ubuntu_nhdx6z/var/games 152K 4.89G 96K /var/games
rpool/ROOT/ubuntu_nhdx6z/var/lib 3.10G 4.89G 2.74G /var/lib
rpool/ROOT/ubuntu_nhdx6z/var/lib/AccountsService 688K 4.89G 104K /var/lib/AccountsService
rpool/ROOT/ubuntu_nhdx6z/var/lib/NetworkManager 1.76M 4.89G 164K /var/lib/NetworkManager
rpool/ROOT/ubuntu_nhdx6z/var/lib/apt 70.1M 4.89G 53.6M /var/lib/apt
rpool/ROOT/ubuntu_nhdx6z/var/lib/dpkg 97.8M 4.89G 39.4M /var/lib/dpkg
rpool/ROOT/ubuntu_nhdx6z/var/log 138M 4.89G 36.4M /var/log
rpool/ROOT/ubuntu_nhdx6z/var/mail 152K 4.89G 96K /var/mail
rpool/ROOT/ubuntu_nhdx6z/var/snap 1.31M 4.89G 240K /var/snap
rpool/ROOT/ubuntu_nhdx6z/var/spool 848K 4.89G 120K /var/spool
rpool/ROOT/ubuntu_nhdx6z/var/www 152K 4.89G 96K /var/www
rpool/ROOT/ubuntu_smztg7 471M 4.89G 3.18G /
rpool/ROOT/ubuntu_smztg7/srv 56K 4.89G 96K /srv
rpool/ROOT/ubuntu_smztg7/usr 144K 4.89G 96K /usr
rpool/ROOT/ubuntu_smztg7/usr/local 144K 4.89G 144K /usr/local
rpool/ROOT/ubuntu_smztg7/var 203M 4.89G 96K /var
rpool/ROOT/ubuntu_smztg7/var/games 112K 4.89G 96K /var/games
rpool/ROOT/ubuntu_smztg7/var/lib 177M 4.89G 2.65G /var/lib
rpool/ROOT/ubuntu_smztg7/var/lib/AccountsService 144K 4.89G 104K /var/lib/AccountsService
rpool/ROOT/ubuntu_smztg7/var/lib/NetworkManager 320K 4.89G 148K /var/lib/NetworkManager
rpool/ROOT/ubuntu_smztg7/var/lib/apt 4.08M 4.89G 53.3M /var/lib/apt
rpool/ROOT/ubuntu_smztg7/var/lib/dpkg 14.0M 4.89G 39.4M /var/lib/dpkg
rpool/ROOT/ubuntu_smztg7/var/log 24.9M 4.89G 15.0M /var/log
rpool/ROOT/ubuntu_smztg7/var/mail 112K 4.89G 96K /var/mail
rpool/ROOT/ubuntu_smztg7/var/snap 212K 4.89G 188K /var/snap
rpool/ROOT/ubuntu_smztg7/var/spool 160K 4.89G 120K /var/spool
rpool/ROOT/ubuntu_smztg7/var/www 112K 4.89G 96K /var/www
rpool/USERDATA 432G 4.89G 96K /
rpool/USERDATA/helper_zthl10 180K 4.89G 124K /home/helper
rpool/USERDATA/root_fv2wpz 316K 4.89G 1.12M /root
rpool/USERDATA/root_mhe8ya 1.78M 4.89G 992K /root
rpool/USERDATA/thomas_fv2wpz 971M 4.89G 278G /home/thomas
rpool/USERDATA/thomas_mhe8ya 431G 4.89G 217G /home/thomas
$ zfs list -o space rpool/USERDATA
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
rpool/USERDATA 4.88G 432G 0B 96K 0B 432G
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 5.8G 0 5.8G 0% /dev
tmpfs 1.2G 2.1M 1.2G 1% /run
rpool/ROOT/ubuntu_nhdx6z 8.8G 3.7G 5.1G 42% /
tmpfs 5.8G 253M 5.6G 5% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
rpool/USERDATA/helper_zthl10 5.1G 128K 5.1G 1% /home/helper
bpool/BOOT/ubuntu_nhdx6z 1.7G 180M 1.5G 11% /boot
rpool/USERDATA/thomas_fv2wpz 284G 279G 5.1G 99% /home/thomas
rpool/USERDATA/root_fv2wpz 5.1G 1.0M 5.1G 1% /root
rpool/ROOT/ubuntu_nhdx6z/srv 5.1G 128K 5.1G 1% /srv
rpool/ROOT/ubuntu_nhdx6z/var/games 5.1G 128K 5.1G 1% /var/games
rpool/ROOT/ubuntu_nhdx6z/var/log 5.2G 29M 5.1G 1% /var/log
rpool/ROOT/ubuntu_nhdx6z/var/lib 7.9G 2.8G 5.1G 35% /var/lib
rpool/ROOT/ubuntu_nhdx6z/usr/local 5.1G 256K 5.1G 1% /usr/local
rpool/ROOT/ubuntu_nhdx6z/var/mail 5.1G 128K 5.1G 1% /var/mail
rpool/ROOT/ubuntu_nhdx6z/var/www 5.1G 128K 5.1G 1% /var/www
/dev/sda1 511M 7.1M 504M 2% /boot/efi
rpool/ROOT/ubuntu_nhdx6z/var/snap 5.1G 256K 5.1G 1% /var/snap
rpool/ROOT/ubuntu_nhdx6z/var/spool 5.1G 128K 5.1G 1% /var/spool
rpool/ROOT/ubuntu_nhdx6z/var/lib/AccountsService 5.1G 128K 5.1G 1% /var/lib/AccountsService
/dev/loop2 62M 62M 0 100% /snap/caprine/36
/dev/loop3 55M 55M 0 100% /snap/core18/1754
/dev/loop4 176M 176M 0 100% /snap/gimp/273
/dev/loop0 9.2M 9.2M 0 100% /snap/canonical-livepatch/95
/dev/loop5 241M 241M 0 100% /snap/gnome-3-34-1804/24
rpool/ROOT/ubuntu_nhdx6z/var/lib/dpkg 5.2G 40M 5.1G 1% /var/lib/dpkg
rpool/ROOT/ubuntu_nhdx6z/var/lib/apt 5.2G 54M 5.1G 2% /var/lib/apt
rpool/ROOT/ubuntu_nhdx6z/var/lib/NetworkManager 5.1G 256K 5.1G 1% /var/lib/NetworkManager
/dev/loop7 174M 174M 0 100% /snap/gimp/252
/dev/loop6 94M 94M 0 100% /snap/core/9066
/dev/loop1 55M 55M 0 100% /snap/core18/1705
/dev/loop8 164M 164M 0 100% /snap/spotify/41
/dev/loop9 256K 256K 0 100% /snap/gtk2-common-themes/9
/dev/loop10 143M 143M 0 100% /snap/slack/23
/dev/loop11 256M 256M 0 100% /snap/gnome-3-34-1804/33
/dev/loop13 63M 63M 0 100% /snap/gtk-common-themes/1506
/dev/loop12 28M 28M 0 100% /snap/snapd/7264
/dev/loop14 50M 50M 0 100% /snap/snap-store/454
/dev/loop15 142M 142M 0 100% /snap/zoom-client/83
/dev/loop16 161M 161M 0 100% /snap/gnome-3-28-1804/116
/dev/loop17 87M 87M 0 100% /snap/shotcut/64
/dev/loop18 291M 291M 0 100% /snap/vlc/1620
/dev/loop19 140M 140M 0 100% /snap/slack/24
/dev/loop20 50M 50M 0 100% /snap/snap-store/433
/dev/loop21 145M 145M 0 100% /snap/zoom-client/84
tmpfs 1.2G 52K 1.2G 1% /run/user/1000
/dev/loop22 64M 64M 0 100% /snap/caprine/37
$ du -h --max-depth=1
25K ./Music
1.0K ./.ssh
1.0K ./Templates
36K ./.gnome
30M ./.thunderbird
52M ./Documents
368M ./snap
8.4M ./.mozilla
203G ./Pictures
302M ./.config
1021M ./.cache
1.0K ./Public
9.5G ./Downloads
101M ./.local
7.0K ./Steam
41G ./Videos
28K ./.gnupg
7.6G ./Desktop
16G ./.steam
17K ./.pki
11K ./.psensor
279G .
$ lsblk -f
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
loop0 squashfs 0 100% /snap/canonical-livepatch/95
loop1 squashfs 0 100% /snap/core18/1705
loop2 squashfs 0 100% /snap/caprine/36
loop3 squashfs 0 100% /snap/core18/1754
loop4 squashfs 0 100% /snap/gimp/273
loop5 squashfs 0 100% /snap/gnome-3-34-1804/24
loop6 squashfs 0 100% /snap/core/9066
loop7 squashfs 0 100% /snap/gimp/252
loop8 squashfs 0 100% /snap/spotify/41
loop9 squashfs 0 100% /snap/gtk2-common-themes/9
loop10 squashfs 0 100% /snap/slack/23
loop11 squashfs 0 100% /snap/gnome-3-34-1804/33
loop12 squashfs 0 100% /snap/snapd/7264
loop13 squashfs 0 100% /snap/gtk-common-themes/1506
loop14 squashfs 0 100% /snap/snap-store/454
loop15 squashfs 0 100% /snap/zoom-client/83
loop16 squashfs 0 100% /snap/gnome-3-28-1804/116
loop17 squashfs 0 100% /snap/shotcut/64
loop18 squashfs 0 100% /snap/vlc/1620
loop19 squashfs 0 100% /snap/slack/24
loop20 squashfs 0 100% /snap/snap-store/433
loop21 squashfs 0 100% /snap/zoom-client/84
loop22 squashfs 0 100% /snap/caprine/37
sda
├─sda1 vfat 105C-066C 503.9M 1% /boot/efi
├─sda2
├─sda5 swap ca95a35a-589b-4127-91d0-ba74f05c7011 [SWAP]
├─sda6 zfs_member bpool 10888252588400514123
└─sda7 zfs_member rpool 8231086401312639965
sr0