Questions[zvol](server)

I have a pair of 480 GB "Datacenter" SSDs (SAMSUNG MZ7LM480HMHQ-00005) that will make up a ZFS pool in a mirroring configuration. The pool's only content will be a ZFS volume (ZVOL) for a virtual machine. However, the ZVOL will only be 400 GB in size, leaving 80 GB of unused space on each SSD.

I'd like the SSDs to make the best use of their unused space, allowing them to use it for reducing the wearing of the flash memory. What would be the best way to achieve this?

1. Simply use the whole SSDs for the pool?

2. Create a 400 GB partition on each SSD and use these for the pool, leaving 80 GB unpartitioned?

3. Try to change the SSDs' Host Protected Area (HPA) so that only 400 GB of capacity will be visible and then use the "shrunk" SSDs for the pool (as in option #1)?

Or maybe there is no difference at all, and the SSD will automatically make good use of all of its flash cells?

If it matters: I'm assuming that the virtual machine will issue TRIM commands and that they reach the host's ZFS and finally the SSDs themselves.

I'm compressing a dd img of a 3TB drive onto a zvol in ZFS for Linux. I enabled compression (lz4) and let it transfer. The pool just consists of one 3TB drive (for now). I am expecting to have 86Gigs more in zfs list than I appear to. Here are some figures:

$ zfs --version
zfs-0.8.3-1ubuntu12
zfs-kmod-0.8.3-1ubuntu12

$ zfs list
NAME                          USED  AVAIL     REFER  MOUNTPOINT
tank                         2.46T   176G       96K  /tank
tank/justin                   100K   176G      100K  /tank/justin
tank/seagate_3tb_01_20_2020  2.46T   176G     2.46T  -

$ zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank  2.72T  2.46T   263G        -         -     0%    90%  1.00x    ONLINE  -

$ zfs get all tank/seagate_3tb_01_20_2020 
NAME                         PROPERTY              VALUE                  SOURCE
tank/seagate_3tb_01_20_2020  type                  volume                 -
tank/seagate_3tb_01_20_2020  creation              Mon Apr 27  0:26 2020  -
tank/seagate_3tb_01_20_2020  used                  2.46T                  -
tank/seagate_3tb_01_20_2020  available             176G                   -
tank/seagate_3tb_01_20_2020  referenced            2.46T                  -
tank/seagate_3tb_01_20_2020  compressratio         1.08x                  -
tank/seagate_3tb_01_20_2020  reservation           none                   default
tank/seagate_3tb_01_20_2020  volsize               3T                     local
tank/seagate_3tb_01_20_2020  volblocksize          8K                     default
tank/seagate_3tb_01_20_2020  checksum              on                     default
tank/seagate_3tb_01_20_2020  compression           lz4                    local
tank/seagate_3tb_01_20_2020  readonly              off                    default
tank/seagate_3tb_01_20_2020  createtxg             10771                  -
tank/seagate_3tb_01_20_2020  copies                1                      default
tank/seagate_3tb_01_20_2020  refreservation        none                   local
tank/seagate_3tb_01_20_2020  guid                  17633099490469485439   -
tank/seagate_3tb_01_20_2020  primarycache          all                    default
tank/seagate_3tb_01_20_2020  secondarycache        all                    default
tank/seagate_3tb_01_20_2020  usedbysnapshots       0B                     -
tank/seagate_3tb_01_20_2020  usedbydataset         2.46T                  -
tank/seagate_3tb_01_20_2020  usedbychildren        0B                     -
tank/seagate_3tb_01_20_2020  usedbyrefreservation  0B                     -
tank/seagate_3tb_01_20_2020  logbias               latency                default
tank/seagate_3tb_01_20_2020  objsetid              906                    -
tank/seagate_3tb_01_20_2020  dedup                 off                    default
tank/seagate_3tb_01_20_2020  mlslabel              none                   default
tank/seagate_3tb_01_20_2020  sync                  standard               default
tank/seagate_3tb_01_20_2020  refcompressratio      1.08x                  -
tank/seagate_3tb_01_20_2020  written               2.46T                  -
tank/seagate_3tb_01_20_2020  logicalused           2.65T                  -
tank/seagate_3tb_01_20_2020  logicalreferenced     2.65T                  -
tank/seagate_3tb_01_20_2020  volmode               default                default
tank/seagate_3tb_01_20_2020  snapshot_limit        none                   default
tank/seagate_3tb_01_20_2020  snapshot_count        none                   default
tank/seagate_3tb_01_20_2020  snapdev               hidden                 default
tank/seagate_3tb_01_20_2020  context               none                   default
tank/seagate_3tb_01_20_2020  fscontext             none                   default
tank/seagate_3tb_01_20_2020  defcontext            none                   default
tank/seagate_3tb_01_20_2020  rootcontext           none                   default
tank/seagate_3tb_01_20_2020  redundant_metadata    all                    default
tank/seagate_3tb_01_20_2020  encryption            off                    default
tank/seagate_3tb_01_20_2020  keylocation           none                   default
tank/seagate_3tb_01_20_2020  keyformat             none                   default
tank/seagate_3tb_01_20_2020  pbkdf2iters           0                      default

I have no snapshots, have disabled reservation and refreservasion, but still have some sort of space discrepancy between the zpool and list.(Yes, I've read the articles, I know they won't be the same, but this is a single disk in the pool, surely ~86G is too much).It seems that I'm not regaining my compression savings to reuse elsewhere (I can only use 176G instead of 263G)

I'm hoping it's a config change I'm missing.

EDIT (zpool get all tank, created with just zpool create tank /dev/sda)

$ zpool get all tank
NAME  PROPERTY                       VALUE                          SOURCE
tank  size                           2.72T                          -
tank  capacity                       90%                            -
tank  altroot                        -                              default
tank  health                         ONLINE                         -
tank  guid                           901113366047988914             -
tank  version                        -                              default
tank  bootfs                         -                              default
tank  delegation                     on                             default
tank  autoreplace                    off                            default
tank  cachefile                      -                              default
tank  failmode                       wait                           default
tank  listsnapshots                  on                             local
tank  autoexpand                     off                            default
tank  dedupditto                     0                              default
tank  dedupratio                     1.00x                          -
tank  free                           263G                           -
tank  allocated                      2.46T                          -
tank  readonly                       off                            -
tank  ashift                         0                              default
tank  comment                        -                              default
tank  expandsize                     -                              -
tank  freeing                        0                              -
tank  fragmentation                  0%                             -
tank  leaked                         0                              -
tank  multihost                      off                            default
tank  checkpoint                     -                              -
tank  load_guid                      9196014585464561985            -
tank  autotrim                       off                            default
tank  feature@async_destroy          enabled                        local
tank  feature@empty_bpobj            active                         local
tank  feature@lz4_compress           active                         local
tank  feature@multi_vdev_crash_dump  enabled                        local
tank  feature@spacemap_histogram     active                         local
tank  feature@enabled_txg            active                         local
tank  feature@hole_birth             active                         local
tank  feature@extensible_dataset     active                         local
tank  feature@embedded_data          active                         local
tank  feature@bookmarks              enabled                        local
tank  feature@filesystem_limits      enabled                        local
tank  feature@large_blocks           enabled                        local
tank  feature@large_dnode            enabled                        local
tank  feature@sha512                 enabled                        local
tank  feature@skein                  enabled                        local
tank  feature@edonr                  enabled                        local
tank  feature@userobj_accounting     active                         local
tank  feature@encryption             enabled                        local
tank  feature@project_quota          active                         local
tank  feature@device_removal         enabled                        local
tank  feature@obsolete_counts        enabled                        local
tank  feature@zpool_checkpoint       enabled                        local
tank  feature@spacemap_v2            active                         local
tank  feature@allocation_classes     enabled                        local
tank  feature@resilver_defer         enabled                        local
tank  feature@bookmark_v2            enabled                        local

I seem to be missing some zvol functionality, but I'm not sure how to phrase it. I had this functionality with these same zpools on an ArchLinux installation. (Migrated the server.)

Ubuntu puts the zvols in /dev/zvol and arch mounted them in /dev. That's okay, but I don't know what happened to the partitions that are created on the zvol itself:

root@nosgoth:/dev/zvol/tank2/vm# fdisk -l tisc
Disk tisc: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes
Disklabel type: dos
Disk identifier: 0x0006b0ad

Device Boot    Start      End  Sectors  Size Id Type
tisc1           2048 37109759 37107712 17.7G 83 Linux
tisc2       37109760 41940991  4831232  2.3G 82 Linux swap / Solaris

On archlinux this would result in devices called tiscPart0 and tiscPart1, and ultimately some /dev/zd64p1 and /dev/zd64p2 Since the pretty names are just a symlink to real devices lrwxrwxrwx 1 root root 13 May 31 21:36 tisc -> ../../../zd64

These things are also mentioned in ZFS administration guides: See the EXT4 on ZVOL section.

I'm assuming there's perhaps a udev configuration? I'm not sure what I'm missing, and searches aren't coming up very helpful. How do I get those partitions to show up?

Is there a way in which ZFS can be prompted to redistribute a given filesystem over the all of the disks in its zpool?

I'm thinking of a scenario where I have a fixed size ZFS volume that's exported as a LUN over FC. The current zpool is small, just two 1TB mirrored disks, and the zvol is 750GB in total. If I were to suddenly expand the size of the zpool to, say, 12 1TB disks, I believe the zvol would still effectively be 'housed' on the first two spindles only.

Given that more spindles = more IOPS, what method could I use to 'redistribute' the zvol over all 12 spindles to take advantage of them?

I've just built a small NAS/SAN box and am doing some testing with it. Configuration is:

• 2GB RAM
• 2x Opteron 265s
• Iwill DK8N
• 5x500GB SATA drives (1 spare, rest mirrored) = 1TB usable space
• Emulex LPC10000DC
• NexentaCore 3.0.2

The first thing I'm testing is COMSTAR performance over fibre-channel. I created a 700g zvol and exposed it over COMSTAR. Write-caching is enabled. My FC initiator is a Windows 7 box with a Q-Logic QLA2340 card in it.

Everything works great, except for write speeds. For some reason, I'm only seeing ~30MB/s, whereas I can peak at about 90MB/s read speeds, using IOMeter. Doing some hunting around, I ran a zpool iostat command and saw the following pattern:

$ zpool iostat tank 1
tank        88.2G   840G      0  5.47K      0  43.4M
tank        88.2G   840G      0  16.1K      0   128M
tank        88.4G   840G      1  6.69K  8.93K  52.6M
tank        88.4G   840G     52  4.89K   110K  38.3M
tank        88.4G   840G    151      0   280K      0
tank        88.4G   840G    112      0   253K      0
tank        88.4G   840G      1  9.89K  16.0K  78.6M
tank        88.4G   840G      0  14.3K      0   113M
tank        88.6G   839G    150  2.99K   318K  20.2M
tank        88.6G   839G     15      0  45.0K      0
tank        88.6G   839G      3      0  17.8K      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0  11.3K      0  89.9M
tank        88.6G   839G      0  13.8K      0   110M
tank        88.8G   839G     90  7.91K   184K  61.0M
tank        88.9G   839G    105     91   220K   140K
tank        88.9G   839G     42      0  84.6K      0
tank        88.9G   839G      0      0      0      0
tank        88.9G   839G      0  6.52K  7.99K  51.8M
tank        88.9G   839G      0  15.6K   1011   124M
tank        88.9G   839G     91  4.30K   193K  31.9M
tank        89.1G   839G      1  9.48K  15.8K  74.7M
tank        89.1G   839G     41      0  99.9K      0
tank        89.1G   839G    166      0   374K      0
tank        89.1G   839G     87      0   162K      0
tank        89.1G   839G      0      0  7.92K      0

It appears to be writing a lot, then stopping.

What's going on here?