I was running FreeBSD 10.0-RELEASE-p4 and migrated to NAS4Free which is FreeBSD version 10.2-RELEASE #0 r286738M: Thu Aug 13 21:32:58 CEST 2015. Now I'm missing free space and I can't delete any files from the pool. Reading files works fine.
All drives are 2 TB. I didn't remember to export the pool before I migrated. According to internet sources exporting and importing pool should free the space but it doesn't. Pool is running version 15. Could upgrade or scrub help?
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da0 ONLINE 0 0 0 block size: 512B configured, 4096B native
da3 ONLINE 0 0 0 block size: 512B configured, 4096B native
da6 ONLINE 0 0 0 block size: 512B configured, 4096B native
da9 ONLINE 0 0 0 block size: 512B configured, 4096B native
da12 ONLINE 0 0 0 block size: 512B configured, 4096B native
da15 ONLINE 0 0 0 block size: 512B configured, 4096B native
raidz2-1 ONLINE 0 0 0
da1 ONLINE 0 0 0 block size: 512B configured, 4096B native
da4 ONLINE 0 0 0 block size: 512B configured, 4096B native
da7 ONLINE 0 0 0 block size: 512B configured, 4096B native
da10 ONLINE 0 0 0 block size: 512B configured, 4096B native
da13 ONLINE 0 0 0 block size: 512B configured, 4096B native
da16 ONLINE 0 0 0 block size: 512B configured, 4096B native
raidz2-2 ONLINE 0 0 0
da2 ONLINE 0 0 0 block size: 512B configured, 4096B native
da5 ONLINE 0 0 0 block size: 512B configured, 4096B native
da8 ONLINE 0 0 0 block size: 512B configured, 4096B native
da11 ONLINE 0 0 0 block size: 512B configured, 4096B native
da14 ONLINE 0 0 0 block size: 512B configured, 4096B native
da17 ONLINE 0 0 0 block size: 512B configured, 4096B native
So: ((6 drives - 2 drives for parity) x 2 TB) x 3 raidz2 = 24 TB which is 21.83 tebibytes usable space minus metadata.
Properties:
# zpool get all
NAME PROPERTY VALUE SOURCE
storage size 32.6T -
storage capacity 97% -
storage altroot - default
storage health ONLINE -
storage guid 14393350251601347202 local
storage version 15 local
storage bootfs - default
storage delegation on default
storage autoreplace off default
storage cachefile - default
storage failmode wait default
storage listsnapshots off default
storage autoexpand off default
storage dedupditto 0 default
storage dedupratio 1.00x -
storage free 696G -
storage allocated 31.9T -
storage readonly off -
storage comment - default
storage expandsize - -
storage freeing 0 local
storage fragmentation - -
storage leaked 0 local
storage feature@async_destroy disabled local
storage feature@empty_bpobj disabled local
storage feature@lz4_compress disabled local
storage feature@multi_vdev_crash_dump disabled local
storage feature@spacemap_histogram disabled local
storage feature@enabled_txg disabled local
storage feature@hole_birth disabled local
storage feature@extensible_dataset disabled local
storage feature@embedded_data disabled local
storage feature@bookmarks disabled local
storage feature@filesystem_limits disabled local
storage feature@large_blocks disabled local
zpool iostat shows free space:
# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
storage 31.9T 696G 0 0 550 540
raidz2 10.8T 63.1G 0 0 162 2
da0 - - 0 0 185 135
da3 - - 0 0 157 135
da6 - - 0 0 158 135
da9 - - 0 0 163 135
da12 - - 0 0 158 135
da15 - - 0 0 149 135
raidz2 10.8T 90.1G 0 0 162 2
da1 - - 0 0 125 101
da4 - - 0 0 102 101
da7 - - 0 0 115 101
da10 - - 0 0 125 101
da13 - - 0 0 100 101
da16 - - 0 0 106 101
raidz2 10.3T 543G 0 0 225 536
da2 - - 0 0 139 243
da5 - - 0 0 133 244
da8 - - 0 0 118 246
da11 - - 0 0 123 246
da14 - - 0 0 123 244
da17 - - 0 0 124 243
---------- ----- ----- ----- ----- ----- -----
zpool list shows free space:
# zpool list storage
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
storage 32.6T 31.9T 696G - - 97% 1.00x ONLINE -
zfs list doesn't show free space:
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
storage 21.3T 0 21.3T /mnt/storage
df doesn't show free space:
# df -ha
Filesystem Size Used Avail Capacity Mounted on
/dev/xmd0 120M 69M 51M 58% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/xmd1 719M 260M 459M 36% /usr/local
procfs 4.0K 4.0K 0B 100% /proc
storage 21T 21T 0B 100% /mnt/storage
/dev/xmd2 248M 3.9M 224M 2% /var
tmpfs 128M 48K 128M 0% /var/tmp
/dev/ada0s1a 743M 151M 592M 20% /cf
There's no snapshots
# zfs list -t snapshot
no datasets available
Datasets:
# zdb -d storage
Dataset mos [META], ID 0, cr_txg 4, 50.0M, 565 objects
Dataset storage [ZPL], ID 16, cr_txg 1, 21.3T, 1111968 objects
Verified large_blocks feature refcount is correct (0)
Deduplication:
# zdb -D storage
All DDTs are empty
camcontrol:
# camcontrol devlist -v
scbus0 on mpt0 bus 0:
<ATA SAMSUNG HD204UI 0001> at scbus0 target 0 lun 0 (pass0,da0)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 1 lun 0 (pass1,da1)
<ATA ST2000DM001-1CH1 CC43> at scbus0 target 2 lun 0 (pass2,da2)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 3 lun 0 (pass3,da3)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 4 lun 0 (pass4,da4)
<ATA ST2000DM001-1CH1 CC43> at scbus0 target 5 lun 0 (pass5,da5)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 6 lun 0 (pass6,da6)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 7 lun 0 (pass7,da7)
<ATA ST2000DM001-1CH1 CC43> at scbus0 target 8 lun 0 (pass8,da8)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 9 lun 0 (pass9,da9)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 10 lun 0 (pass10,da10)
<ATA ST2000DM001-1CH1 CC43> at scbus0 target 11 lun 0 (pass11,da11)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 12 lun 0 (pass12,da12)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 13 lun 0 (pass13,da13)
<ATA ST2000DM001-1CH1 CC43> at scbus0 target 14 lun 0 (pass14,da14)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 15 lun 0 (pass15,da15)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 16 lun 0 (pass16,da16)
<ATA ST2000DM001-1CH1 CC43> at scbus0 target 17 lun 0 (pass17,da17)
<HP HP SAS EXP Card 2.06> at scbus0 target 18 lun 0 (pass18,ses0)
<> at scbus0 target -1 lun ffffffff ()
scbus1 on ata2 bus 0:
<> at scbus1 target -1 lun ffffffff ()
scbus2 on ata3 bus 0:
<> at scbus2 target -1 lun ffffffff ()
scbus3 on ata4 bus 0:
<KINGSTON SV300S37A60G 525ABBF0> at scbus3 target 0 lun 0 (ada0,pass19)
<> at scbus3 target -1 lun ffffffff ()
scbus4 on ata5 bus 0:
<> at scbus4 target -1 lun ffffffff ()
scbus-1 on xpt0 bus 0:
<> at scbus-1 target -1 lun ffffffff (xpt0)
FreeBSD 10.2 added tunable,
vfs.zfs.spa_slop_shift, which controls how much space is reserved by default.Source: https://www.freebsd.org/releases/10.2R/relnotes.html#storage-zfs