Ryan Babchishin's questions

I'm having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED package provided be Mellanox (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz). My cards are Mellanox 10Gbe and are RoCE v1 enabled.

Works with Inbox drivers/software but not no much with latest OFED

I managed to get NFS working with RoCE by following the docs on this site using the Inbox drivers/software (included with Ubuntu 16.04). I was having some little issues and I know the Ubuntu stuff is quite out of date so I wanted to install the latest OFED/mlx4 drivers, etc... as per recommendations on mellanox.com. So I did that. All went as planned. IP functionality is all there and RDMA tools/tests all work. Everything seems to work great. Except one thing.

The svcrdma and xprtrdma modules won't load. So no RDMA support for NFS for me. I get the following errors. I also get the same error if I install only the latest mlx4 driver from the Mellanox site and leave the rest of the packages alone.

I have a feeling this can be resolved somehow - like by recompiling kernel modules and such but that is over my head at the moment. Or maybe I just messed something up (crossing fingers)? Can anyone help?

Someone commented on this Mellanox community article that they had the same issue with Ubuntu 14.04: https://community.mellanox.com/docs/DOC-2132 According to the same document, it should work just fine with CentOS 7. What's the difference?

The end result I want is to have the latest driver and software (preferably) working on Ubuntu 16.04 with NFS over RoCE. If not the latest OFED package, at least the latest mlx4 driver. I read somewhere that newer kernel versions will have updated drivers and RDMA code (I forgot most of what I read). If this goes nowhere, my answer may have to be to wait for a newer Ubuntu release.

Thanks

Error messages when loading modules

NFS server:

# modprobe svcrdma 
modprobe: ERROR: could not insert 'rpcrdma': Invalid argument 

dmesg errors:

[105699.696980] rpcrdma: Unknown symbol rdma_event_msg (err 0) 
[105699.697056] rpcrdma: disagrees about version of symbol ib_create_cq 
[105699.697059] rpcrdma: Unknown symbol ib_create_cq (err -22) 
[105699.697069] rpcrdma: disagrees about version of symbol rdma_resolve_addr 
[105699.697071] rpcrdma: Unknown symbol rdma_resolve_addr (err -22) 
[105699.697183] rpcrdma: Unknown symbol ib_event_msg (err 0) 
[105699.697213] rpcrdma: disagrees about version of symbol ib_dereg_mr 
[105699.697215] rpcrdma: Unknown symbol ib_dereg_mr (err -22) 
[105699.697224] rpcrdma: disagrees about version of symbol ib_query_qp 
[105699.697226] rpcrdma: Unknown symbol ib_query_qp (err -22) 
[105699.697236] rpcrdma: disagrees about version of symbol rdma_disconnect 
[105699.697238] rpcrdma: Unknown symbol rdma_disconnect (err -22) 
[105699.697245] rpcrdma: disagrees about version of symbol ib_alloc_fmr 
[105699.697247] rpcrdma: Unknown symbol ib_alloc_fmr (err -22) 
[105699.697294] rpcrdma: disagrees about version of symbol ib_dealloc_fmr 
[105699.697295] rpcrdma: Unknown symbol ib_dealloc_fmr (err -22) 
[105699.697301] rpcrdma: disagrees about version of symbol rdma_resolve_route 
[105699.697303] rpcrdma: Unknown symbol rdma_resolve_route (err -22) 
[105699.697398] rpcrdma: disagrees about version of symbol rdma_bind_addr 
[105699.697400] rpcrdma: Unknown symbol rdma_bind_addr (err -22) 
[105699.697441] rpcrdma: disagrees about version of symbol rdma_create_qp 
[105699.697443] rpcrdma: Unknown symbol rdma_create_qp (err -22) 
[105699.697479] rpcrdma: Unknown symbol ib_map_mr_sg (err 0) 
[105699.697487] rpcrdma: disagrees about version of symbol ib_destroy_cq 
[105699.697489] rpcrdma: Unknown symbol ib_destroy_cq (err -22) 
[105699.697494] rpcrdma: disagrees about version of symbol rdma_create_id 
[105699.697496] rpcrdma: Unknown symbol rdma_create_id (err -22) 
[105699.697582] rpcrdma: disagrees about version of symbol rdma_listen 
[105699.697584] rpcrdma: Unknown symbol rdma_listen (err -22) 
[105699.697587] rpcrdma: disagrees about version of symbol rdma_destroy_qp 
[105699.697589] rpcrdma: Unknown symbol rdma_destroy_qp (err -22) 
[105699.697597] rpcrdma: disagrees about version of symbol ib_query_device 
[105699.697599] rpcrdma: Unknown symbol ib_query_device (err -22) 
[105699.697606] rpcrdma: disagrees about version of symbol ib_get_dma_mr 
[105699.697607] rpcrdma: Unknown symbol ib_get_dma_mr (err -22) 
[105699.697617] rpcrdma: disagrees about version of symbol ib_alloc_pd 
[105699.697618] rpcrdma: Unknown symbol ib_alloc_pd (err -22) 
[105699.697673] rpcrdma: Unknown symbol ib_alloc_mr (err 0) 
[105699.697734] rpcrdma: disagrees about version of symbol rdma_connect 
[105699.697736] rpcrdma: Unknown symbol rdma_connect (err -22) 
[105699.697769] rpcrdma: Unknown symbol ib_wc_status_msg (err 0) 
[105699.697842] rpcrdma: disagrees about version of symbol rdma_destroy_id 
[105699.697844] rpcrdma: Unknown symbol rdma_destroy_id (err -22) 
[105699.697872] rpcrdma: disagrees about version of symbol rdma_accept 
[105699.697874] rpcrdma: Unknown symbol rdma_accept (err -22) 
[105699.697882] rpcrdma: disagrees about version of symbol ib_destroy_qp 
[105699.697883] rpcrdma: Unknown symbol ib_destroy_qp (err -22) 
[105699.697964] rpcrdma: disagrees about version of symbol ib_dealloc_pd 
[105699.697965] rpcrdma: Unknown symbol ib_dealloc_pd (err -22)

NFS client:

# modprobe xprtrdma          
modprobe: ERROR: could not insert 'rpcrdma': Invalid argument

dmesg errors:

[106055.692454] rpcrdma: Unknown symbol rdma_event_msg (err 0) 
[106055.692480] rpcrdma: disagrees about version of symbol ib_create_cq 
[106055.692481] rpcrdma: Unknown symbol ib_create_cq (err -22) 
[106055.692484] rpcrdma: disagrees about version of symbol rdma_resolve_addr 
[106055.692485] rpcrdma: Unknown symbol rdma_resolve_addr (err -22) 
[106055.692520] rpcrdma: Unknown symbol ib_event_msg (err 0) 
[106055.692529] rpcrdma: disagrees about version of symbol ib_dereg_mr 
[106055.692530] rpcrdma: Unknown symbol ib_dereg_mr (err -22) 
[106055.692532] rpcrdma: disagrees about version of symbol ib_query_qp 
[106055.692533] rpcrdma: Unknown symbol ib_query_qp (err -22) 
[106055.692536] rpcrdma: disagrees about version of symbol rdma_disconnect 
[106055.692536] rpcrdma: Unknown symbol rdma_disconnect (err -22) 
[106055.692538] rpcrdma: disagrees about version of symbol ib_alloc_fmr 
[106055.692539] rpcrdma: Unknown symbol ib_alloc_fmr (err -22) 
[106055.692552] rpcrdma: disagrees about version of symbol ib_dealloc_fmr 
[106055.692553] rpcrdma: Unknown symbol ib_dealloc_fmr (err -22) 
[106055.692554] rpcrdma: disagrees about version of symbol rdma_resolve_route 
[106055.692555] rpcrdma: Unknown symbol rdma_resolve_route (err -22) 
[106055.692565] rpcrdma: disagrees about version of symbol rdma_bind_addr 
[106055.692565] rpcrdma: Unknown symbol rdma_bind_addr (err -22) 
[106055.692573] rpcrdma: disagrees about version of symbol rdma_create_qp 
[106055.692574] rpcrdma: Unknown symbol rdma_create_qp (err -22) 
[106055.692583] rpcrdma: Unknown symbol ib_map_mr_sg (err 0) 
[106055.692585] rpcrdma: disagrees about version of symbol ib_destroy_cq 
[106055.692585] rpcrdma: Unknown symbol ib_destroy_cq (err -22) 
[106055.692587] rpcrdma: disagrees about version of symbol rdma_create_id 
[106055.692587] rpcrdma: Unknown symbol rdma_create_id (err -22) 
[106055.692613] rpcrdma: disagrees about version of symbol rdma_listen 
[106055.692614] rpcrdma: Unknown symbol rdma_listen (err -22) 
[106055.692615] rpcrdma: disagrees about version of symbol rdma_destroy_qp 
[106055.692615] rpcrdma: Unknown symbol rdma_destroy_qp (err -22) 
[106055.692617] rpcrdma: disagrees about version of symbol ib_query_device 
[106055.692618] rpcrdma: Unknown symbol ib_query_device (err -22) 
[106055.692619] rpcrdma: disagrees about version of symbol ib_get_dma_mr 
[106055.692620] rpcrdma: Unknown symbol ib_get_dma_mr (err -22) 
[106055.692622] rpcrdma: disagrees about version of symbol ib_alloc_pd 
[106055.692623] rpcrdma: Unknown symbol ib_alloc_pd (err -22) 
[106055.692638] rpcrdma: Unknown symbol ib_alloc_mr (err 0) 
[106055.692657] rpcrdma: disagrees about version of symbol rdma_connect 
[106055.692658] rpcrdma: Unknown symbol rdma_connect (err -22) 
[106055.692668] rpcrdma: Unknown symbol ib_wc_status_msg (err 0) 
[106055.692690] rpcrdma: disagrees about version of symbol rdma_destroy_id 
[106055.692690] rpcrdma: Unknown symbol rdma_destroy_id (err -22) 
[106055.692698] rpcrdma: disagrees about version of symbol rdma_accept 
[106055.692699] rpcrdma: Unknown symbol rdma_accept (err -22) 
[106055.692701] rpcrdma: disagrees about version of symbol ib_destroy_qp 
[106055.692701] rpcrdma: Unknown symbol ib_destroy_qp (err -22) 
[106055.692724] rpcrdma: disagrees about version of symbol ib_dealloc_pd 
[106055.692725] rpcrdma: Unknown symbol ib_dealloc_pd (err -22)

My problem with ZFS on Ubuntu Xenial (16.04)

I've been running ZFS for years from the package maintainer's zfsonlinux.org site. Things have been good. But recently I upgraded to Ubuntu 16.04 when it came out. I was excited that it had ZFS built in. And so far everything is working great except one thing.

zfs share -a and zfs share filesystem don't work and because of that, whenever I reboot the system, zfs-share.service does not enable my SMB shares configured in ZFS. I did not have this problem with Debian 8.x or older versions of Ubuntu with the packages from zfsonlinux.org.

I've submitted this as a bug with ZoL and Ubuntu. I have no idea how far that will go. I don't know if it's a bug at all or a problem with Ubuntu, or me. So I'm also hoping someone here will have an idea or can confirm this as a problem they're having too.

ZoL: https://github.com/zfsonlinux/zfs/issues/4999

Ubuntu: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1615405

See at the bottom for my actual question.

Debugging info

No shares! I verified on several client computers too - that's how the problem was discovered.

root@igor:~# net usershare list
root@igor:~# 

Systemd zfs-share.status after a fresh reboot seems fine

root@igor:~# systemctl status zfs-share
● zfs-share.service - LSB: Network share OpenZFS datasets.
   Loaded: loaded (/etc/init.d/zfs-share; static; vendor preset: enabled)
   Active: active (exited) since Fri 2016-08-19 11:06:32 EDT; 4min 56s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6432 ExecStart=/etc/init.d/zfs-share start (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0

Aug 19 11:06:23 igor systemd[1]: Starting LSB: Network share OpenZFS datasets....
Aug 19 11:06:25 igor zfs-share[6432]:  * Sharing OpenZFS filesystems
Aug 19 11:06:32 igor zfs-share[6432]:    ...done.
Aug 19 11:06:32 igor systemd[1]: Started LSB: Network share OpenZFS datasets..

Systemd smbd.service after a fresh reboot seems fine

● smbd.service - LSB: start Samba SMB/CIFS daemon (smbd)
   Loaded: loaded (/etc/init.d/smbd; bad; vendor preset: enabled)
   Active: active (running) since Fri 2016-08-19 11:27:44 EDT; 3min 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6629 ExecStart=/etc/init.d/smbd start (code=exited, status=0/SUCCESS)
    Tasks: 4
   Memory: 6.2M
      CPU: 263ms
   CGroup: /system.slice/smbd.service
           ├─6719 /usr/sbin/smbd -D
           ├─6720 /usr/sbin/smbd -D
           └─6743 /usr/sbin/smbd -D

Aug 19 11:27:42 igor systemd[1]: Starting LSB: start Samba SMB/CIFS daemon (smbd)...
Aug 19 11:27:44 igor smbd[6629]:  * Starting SMB/CIFS daemon smbd
Aug 19 11:27:44 igor smbd[6629]:    ...done.
Aug 19 11:27:44 igor systemd[1]: Started LSB: start Samba SMB/CIFS daemon (smbd).
Aug 19 11:27:46 igor smbd[6844]: pam_unix(samba:session): session opened for user rbabchis by (uid=0)

Datasets are configured properly

root@igor:~# zfs get sharesmb pool2/Media
NAME         PROPERTY  VALUE     SOURCE
pool2/Media  sharesmb  on        local
root@igor:~# zfs get sharesmb pool2/home
NAME        PROPERTY  VALUE     SOURCE
pool2/home  sharesmb  on        local

Even when I share manually nothing happens

root@igor:~# zfs share -a
root@igor:~# net usershare list
root@igor:~# ls /var/lib/samba/usershares/
root@igor:~#

Already shared message when I share a specific filesystem

root@igor:~# zfs share pool2/home
cannot share 'pool2/home': filesystem already shared

ZFS code uses /etc/dfs/sharetab but Linux/Samba don't (after boot) https://github.com/zfsonlinux/zfs/issues/190

root@igor:~# cat /etc/dfs/sharetab 
/pool2/home     -       smb     on
/pool2/Media    -       smb     on

strace -y zfs share -a It looks like ZFS could be making no attempt to share the datasets because it finds them in /etc/dfs/sharetab, but I could be misinterpreting that.

root@igor:~# strace -y zfs share -a
execve("/sbin/zfs", ["zfs", "share", "-a"], [/* 21 vars */]) = 0
brk(NULL)                               = 0x2372000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f100dfdf000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3</etc/ld.so.cache>
fstat(3</etc/ld.so.cache>, {st_mode=S_IFREG|0644, st_size=128650, ...}) = 0
mmap(NULL, 128650, PROT_READ, MAP_PRIVATE, 3</etc/ld.so.cache>, 0) = 0x7f100dfbc000
close(3</etc/ld.so.cache>)              = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libnvpair.so.1", O_RDONLY|O_CLOEXEC) = 3</usr/lib/libnvpair.so.1.0.1>
read(3</usr/lib/libnvpair.so.1.0.1>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@L\0\0\0\0\0\0"..., 832) = 832
fstat(3</usr/lib/libnvpair.so.1.0.1>, {st_mode=S_IFREG|0644, st_size=80896, ...}) = 0
mmap(NULL, 2175984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/libnvpair.so.1.0.1>, 0) = 0x7f100dba4000
mprotect(0x7f100dbb7000, 2093056, PROT_NONE) = 0
mmap(0x7f100ddb6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/libnvpair.so.1.0.1>, 0x12000) = 0x7f100ddb6000
close(3</usr/lib/libnvpair.so.1.0.1>)   = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libuutil.so.1", O_RDONLY|O_CLOEXEC) = 3</usr/lib/libuutil.so.1.0.1>
read(3</usr/lib/libuutil.so.1.0.1>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340V\0\0\0\0\0\0"..., 832) = 832
fstat(3</usr/lib/libuutil.so.1.0.1>, {st_mode=S_IFREG|0644, st_size=73648, ...}) = 0
mmap(NULL, 2173272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/libuutil.so.1.0.1>, 0) = 0x7f100d98c000
mprotect(0x7f100d99d000, 2093056, PROT_NONE) = 0
mmap(0x7f100db9c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/libuutil.so.1.0.1>, 0x10000) = 0x7f100db9c000
mmap(0x7f100db9e000, 2392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f100db9e000
close(3</usr/lib/libuutil.so.1.0.1>)    = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libzpool.so.2", O_RDONLY|O_CLOEXEC) = 3</usr/lib/libzpool.so.2.0.0>
read(3</usr/lib/libzpool.so.2.0.0>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320}\2\0\0\0\0\0"..., 832) = 832
fstat(3</usr/lib/libzpool.so.2.0.0>, {st_mode=S_IFREG|0644, st_size=1281576, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f100dfde000
mmap(NULL, 4839192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/libzpool.so.2.0.0>, 0) = 0x7f100d4ec000
mprotect(0x7f100d61d000, 2097152, PROT_NONE) = 0
mmap(0x7f100d81d000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/libzpool.so.2.0.0>, 0x131000) = 0x7f100d81d000
mmap(0x7f100d825000, 1459992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f100d825000
close(3</usr/lib/libzpool.so.2.0.0>)    = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libzfs.so.2", O_RDONLY|O_CLOEXEC) = 3</usr/lib/libzfs.so.2.0.0>
read(3</usr/lib/libzfs.so.2.0.0>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\252\0\0\0\0\0\0"..., 832) = 832
fstat(3</usr/lib/libzfs.so.2.0.0>, {st_mode=S_IFREG|0644, st_size=275968, ...}) = 0
mmap(NULL, 2371208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/libzfs.so.2.0.0>, 0) = 0x7f100d2a4000
mprotect(0x7f100d2e6000, 2093056, PROT_NONE) = 0
mmap(0x7f100d4e5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/libzfs.so.2.0.0>, 0x41000) = 0x7f100d4e5000
close(3</usr/lib/libzfs.so.2.0.0>)      = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libzfs_core.so.1", O_RDONLY|O_CLOEXEC) = 3</usr/lib/libzfs_core.so.1.0.0>
read(3</usr/lib/libzfs_core.so.1.0.0>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\21\0\0\0\0\0\0"..., 832) = 832
fstat(3</usr/lib/libzfs_core.so.1.0.0>, {st_mode=S_IFREG|0644, st_size=14544, ...}) = 0
mmap(NULL, 2109840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</usr/lib/libzfs_core.so.1.0.0>, 0) = 0x7f100d09c000
mprotect(0x7f100d09f000, 2093056, PROT_NONE) = 0
mmap(0x7f100d29e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</usr/lib/libzfs_core.so.1.0.0>, 0x2000) = 0x7f100d29e000
close(3</usr/lib/libzfs_core.so.1.0.0>) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3</lib/x86_64-linux-gnu/libpthread-2.23.so>
read(3</lib/x86_64-linux-gnu/libpthread-2.23.so>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360`\0\0\0\0\0\0"..., 832) = 832
fstat(3</lib/x86_64-linux-gnu/libpthread-2.23.so>, {st_mode=S_IFREG|0755, st_size=138744, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f100dfdd000
mmap(NULL, 2212904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libpthread-2.23.so>, 0) = 0x7f100ce7c000
mprotect(0x7f100ce94000, 2093056, PROT_NONE) = 0
mmap(0x7f100d093000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libpthread-2.23.so>, 0x17000) = 0x7f100d093000
mmap(0x7f100d095000, 13352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f100d095000
close(3</lib/x86_64-linux-gnu/libpthread-2.23.so>) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3</lib/x86_64-linux-gnu/libc-2.23.so>
read(3</lib/x86_64-linux-gnu/libc-2.23.so>, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
fstat(3</lib/x86_64-linux-gnu/libc-2.23.so>, {st_mode=S_IFREG|0755, st_size=1864888, ...}) = 0
mmap(NULL, 3967488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libc-2.23.so>, 0) = 0x7f100caac000
mprotect(0x7f100cc6c000, 2093056, PROT_NONE) = 0
mmap(0x7f100ce6b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libc-2.23.so>, 0x1bf000) = 0x7f100ce6b000
mmap(0x7f100ce71000, 14848, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f100ce71000
close(3</lib/x86_64-linux-gnu/libc-2.23.so>) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3</lib/x86_64-linux-gnu/librt-2.23.so>
read(3</lib/x86_64-linux-gnu/librt-2.23.so>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0!\0\0\0\0\0\0"..., 832) = 832
fstat(3</lib/x86_64-linux-gnu/librt-2.23.so>, {st_mode=S_IFREG|0644, st_size=31712, ...}) = 0
mmap(NULL, 2128832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/librt-2.23.so>, 0) = 0x7f100c8a4000
mprotect(0x7f100c8ab000, 2093056, PROT_NONE) = 0
mmap(0x7f100caaa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/librt-2.23.so>, 0x6000) = 0x7f100caaa000
close(3</lib/x86_64-linux-gnu/librt-2.23.so>) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3</lib/x86_64-linux-gnu/libuuid.so.1.3.0>
read(3</lib/x86_64-linux-gnu/libuuid.so.1.3.0>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\25\0\0\0\0\0\0"..., 832) = 832
fstat(3</lib/x86_64-linux-gnu/libuuid.so.1.3.0>, {st_mode=S_IFREG|0644, st_size=18976, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f100dfdc000
mmap(NULL, 2113904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libuuid.so.1.3.0>, 0) = 0x7f100c69c000
mprotect(0x7f100c6a0000, 2093056, PROT_NONE) = 0
mmap(0x7f100c89f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libuuid.so.1.3.0>, 0x3000) = 0x7f100c89f000
close(3</lib/x86_64-linux-gnu/libuuid.so.1.3.0>) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3</lib/x86_64-linux-gnu/libz.so.1.2.8>
read(3</lib/x86_64-linux-gnu/libz.so.1.2.8>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\35\0\0\0\0\0\0"..., 832) = 832
fstat(3</lib/x86_64-linux-gnu/libz.so.1.2.8>, {st_mode=S_IFREG|0644, st_size=104824, ...}) = 0
mmap(NULL, 2199880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libz.so.1.2.8>, 0) = 0x7f100c47c000
mprotect(0x7f100c495000, 2093056, PROT_NONE) = 0
mmap(0x7f100c694000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libz.so.1.2.8>, 0x18000) = 0x7f100c694000
close(3</lib/x86_64-linux-gnu/libz.so.1.2.8>) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3</lib/x86_64-linux-gnu/libm-2.23.so>
read(3</lib/x86_64-linux-gnu/libm-2.23.so>, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0V\0\0\0\0\0\0"..., 832) = 832
fstat(3</lib/x86_64-linux-gnu/libm-2.23.so>, {st_mode=S_IFREG|0644, st_size=1088952, ...}) = 0
mmap(NULL, 3178744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libm-2.23.so>, 0) = 0x7f100c16c000
mprotect(0x7f100c274000, 2093056, PROT_NONE) = 0
mmap(0x7f100c473000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libm-2.23.so>, 0x107000) = 0x7f100c473000
close(3</lib/x86_64-linux-gnu/libm-2.23.so>) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3</lib/x86_64-linux-gnu/libblkid.so.1.1.0>
read(3</lib/x86_64-linux-gnu/libblkid.so.1.1.0>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\201\0\0\0\0\0\0"..., 832) = 832
fstat(3</lib/x86_64-linux-gnu/libblkid.so.1.1.0>, {st_mode=S_IFREG|0644, st_size=262408, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f100dfbb000
mmap(NULL, 2361576, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libblkid.so.1.1.0>, 0) = 0x7f100bf24000
mprotect(0x7f100bf60000, 2093056, PROT_NONE) = 0
mmap(0x7f100c15f000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libblkid.so.1.1.0>, 0x3b000) = 0x7f100c15f000
mmap(0x7f100c164000, 2280, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f100c164000
close(3</lib/x86_64-linux-gnu/libblkid.so.1.1.0>) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f100dfba000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f100dfb7000
arch_prctl(ARCH_SET_FS, 0x7f100dfb8780) = 0
mprotect(0x7f100ce6b000, 16384, PROT_READ) = 0
mprotect(0x7f100c89f000, 4096, PROT_READ) = 0
mprotect(0x7f100c15f000, 16384, PROT_READ) = 0
mprotect(0x7f100c473000, 4096, PROT_READ) = 0
mprotect(0x7f100c694000, 4096, PROT_READ) = 0
mprotect(0x7f100d093000, 4096, PROT_READ) = 0
mprotect(0x7f100caaa000, 4096, PROT_READ) = 0
mprotect(0x7f100db9c000, 4096, PROT_READ) = 0
mprotect(0x7f100ddb6000, 4096, PROT_READ) = 0
mprotect(0x7f100d29e000, 4096, PROT_READ) = 0
mprotect(0x7f100d81d000, 8192, PROT_READ) = 0
mprotect(0x7f100d4e5000, 4096, PROT_READ) = 0
mprotect(0x617000, 4096, PROT_READ)     = 0
mprotect(0x7f100dfe1000, 4096, PROT_READ) = 0
munmap(0x7f100dfbc000, 128650)          = 0
set_tid_address(0x7f100dfb8a50)         = 7717
set_robust_list(0x7f100dfb8a60, 24)     = 0
rt_sigaction(SIGRTMIN, {0x7f100ce81b90, [], SA_RESTORER|SA_SIGINFO, 0x7f100ce8d3d0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7f100ce81c20, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f100ce8d3d0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(NULL)                               = 0x2372000
brk(0x2393000)                          = 0x2393000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3</usr/lib/locale/locale-archive>
fstat(3</usr/lib/locale/locale-archive>, {st_mode=S_IFREG|0644, st_size=2977184, ...}) = 0
mmap(NULL, 2977184, PROT_READ, MAP_PRIVATE, 3</usr/lib/locale/locale-archive>, 0) = 0x7f100bc4c000
close(3</usr/lib/locale/locale-archive>) = 0
access("/sys/module/zfs", F_OK)         = 0
access("/sys/module/zfs", F_OK)         = 0
open("/dev/zfs", O_RDWR)                = 3</dev/zfs>
close(3</dev/zfs>)                      = 0
open("/dev/zfs", O_RDWR)                = 3</dev/zfs>
open("/etc/mtab", O_RDONLY)             = 4</proc/7717/mounts>
open("/etc/dfs/sharetab", O_RDONLY)     = 5</etc/dfs/sharetab>
open("/dev/zfs", O_RDWR)                = 6</dev/zfs>
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x04, 0x00), 0x7ffc53b06cc0) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffc53b06cd0) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x05, 0x00), 0x7ffc53b03680) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = -1 ENOMEM (Cannot allocate memory)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b03660) = -1 ESRCH (No such process)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b03660) = -1 ESRCH (No such process)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = -1 ENOMEM (Cannot allocate memory)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = -1 ENOMEM (Cannot allocate memory)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = -1 ENOMEM (Cannot allocate memory)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b03660) = -1 ESRCH (No such process)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = -1 ENOMEM (Cannot allocate memory)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = -1 ESRCH (No such process)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffc53b06cd0) = 0
brk(0x23bf000)                          = 0x23bf000
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x05, 0x00), 0x7ffc53b03680) = 0
brk(0x23af000)                          = 0x23af000
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b03660) = -1 ESRCH (No such process)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = -1 ENOMEM (Cannot allocate memory)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b03660) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b00030) = -1 ESRCH (No such process)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b03660) = -1 ESRCH (No such process)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x14, 0x00), 0x7ffc53b06c90) = -1 ESRCH (No such process)
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x27, 0x00), 0x7ffc53b038e0) = 0
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x27, 0x00), 0x7ffc53b038e0) = 0
open("/etc/mtab", O_RDONLY)             = 7</proc/7717/mounts>
dup3(7</proc/7717/mounts>, 4</proc/7717/mounts>, 0) = 4</proc/7717/mounts>
close(7</proc/7717/mounts>)             = 0
fstat(4</proc/7717/mounts>, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(4</proc/7717/mounts>, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024
read(4</proc/7717/mounts>, "0 0\ncgroup /sys/fs/cgroup/pids c"..., 1024) = 1024
read(4</proc/7717/mounts>, " mqueue rw,relatime 0 0\ndebugfs "..., 1024) = 1024
read(4</proc/7717/mounts>, " 0 0\nhugetlb /run/lxcfs/controll"..., 1024) = 1024
read(4</proc/7717/mounts>, "00,uid=127,gid=136 0 0\ntmpfs /ru"..., 1024) = 99
read(4</proc/7717/mounts>, "", 1024)    = 0
fstat(5</etc/dfs/sharetab>, {st_mode=S_IFREG|0600, st_size=43, ...}) = 0
lseek(5</etc/dfs/sharetab>, 0, SEEK_SET) = 0
read(5</etc/dfs/sharetab>, "/pool2/home\t-\tsmb\ton\n/pool2/Medi"..., 4096) = 43
read(5</etc/dfs/sharetab>, "", 4096)    = 0
lseek(5</etc/dfs/sharetab>, 0, SEEK_SET) = 0
read(5</etc/dfs/sharetab>, "/pool2/home\t-\tsmb\ton\n/pool2/Medi"..., 4096) = 43
lseek(5</etc/dfs/sharetab>, 43, SEEK_SET) = 43
read(5</etc/dfs/sharetab>, "", 4096)    = 0
lseek(5</etc/dfs/sharetab>, 0, SEEK_SET) = 0
read(5</etc/dfs/sharetab>, "/pool2/home\t-\tsmb\ton\n/pool2/Medi"..., 4096) = 43
ioctl(3</dev/zfs>, _IOC(0, 0x5a, 0x3f, 0x00), 0x7ffc53b06dd0) = -1 EPERM (Operation not permitted)
close(3</dev/zfs>)                      = 0
close(4</proc/7717/mounts>)             = 0
close(5</etc/dfs/sharetab>)             = 0
close(6</dev/zfs>)                      = 0
exit_group(0)                           = ?
+++ exited with 0 +++

The only way to get the datasets shared is to reset the sharesmb property to on

root@igor:~# zfs set sharesmb=on pool2/home
root@igor:~# net usershare list
pool2_home
root@igor:~# zfs set sharesmb=on pool2/Media
root@igor:~# net usershare list
pool2_home
pool2_Media

My question

I know I can fix this by adding these entries in /etc/rc.local

zfs set sharesmb=on pool2/Main
zfs set sharesmb=on pool2/home

And I will. But does anyone know why zfs share -a doesn't work as intended (as it always did)? Has anyone else experienced this? Perhaps it affects all Ubuntu Xenial installs or perhaps it's just me. Even if there's no fix and it's a bug, I'd like to know.

Thanks!

I have a related question about this problem, but it got too complicated and too big, so I decided I should split up the issue into NFS and local issues. I have also tried asking about this on the zfs-discuss mailing list without much success.

Slow copying between NFS/CIFS directories on same server

Outline: How I'm setup and what I'm expecting

1. I have a ZFS pool with 4 disks. 2TB RED configured as 2 mirrors that are striped (RAID 10). On Linux, zfsonlinux. There are no cache or log devices.
2. Data is balanced across mirrors (important for ZFS)
3. Each disk can read (raw w/dd) at 147MB/sec in parallel, giving a combined throughput of 588MB/sec.
4. I expect about 115MB/sec write, 138MB/sec read and 50MB/sec rewrite of sequential data from each disk, based on benchmarks of a similar 4TB RED disk. I expect no less than 100MB/sec read or write, since any disk can do that these days.
5. I thought I'd see 100% IO utilization on all 4 disks when under load reading or writing sequential data. And that the disks would be putting out over 100MB/sec while at 100% utilization.
6. I thought the pool would give me around 2x write, 2x rewrite, and 4x read performance over a single disk - am I wrong?
7. NEW I thought a ext4 zvol on the same pool would be about the same speed as ZFS

What I actually get

I find the read performance of the pool is not nearly as high as I expected

bonnie++ benchmark on pool from a few days ago

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
igor            63G    99  99 232132  47 118787  27   336  97 257072  22  92.7   6

bonnie++ on a single 4TB RED drive on it's own in a zpool

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
igor            63G   101  99 115288  30 49781  14   326  97 138250  13 111.6   8

According to this the read and rewrite speeds are appropriate based on the results from a single 4TB RED drive (they are double). However, the read speed I was expecting would have been about 550MB/sec (4x the speed of the 4TB drive) and I would at least hope for around 400MB/sec. Instead I am seeing around 260MB/sec

bonnie++ on the pool from just now, while gathering the below information. Not quite the same as before, and nothing has changed.

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
igor            63G   103  99 207518  43 108810  24   342  98 302350  26 256.4  18

zpool iostat during write. Seems OK to me.

                                                 capacity     operations    bandwidth
pool                                          alloc   free   read  write   read  write
--------------------------------------------  -----  -----  -----  -----  -----  -----
pool2                                         1.23T  2.39T      0  1.89K  1.60K   238M
  mirror                                       631G  1.20T      0    979  1.60K   120M
    ata-WDC_WD20EFRX-68AX9N0_WD-WMC300004469      -      -      0   1007  1.60K   124M
    ata-WDC_WD20EFRX-68EUZN0_WD-WCC4MLK57MVX      -      -      0    975      0   120M
  mirror                                       631G  1.20T      0    953      0   117M
    ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0429536      -      -      0  1.01K      0   128M
    ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M0VYKFCE      -      -      0    953      0   117M

zpool iostat during rewrite. Seems ok to me, I think.

                                                 capacity     operations    bandwidth
pool                                          alloc   free   read  write   read  write
--------------------------------------------  -----  -----  -----  -----  -----  -----
pool2                                         1.27T  2.35T   1015    923   125M   101M
  mirror                                       651G  1.18T    505    465  62.2M  51.8M
    ata-WDC_WD20EFRX-68AX9N0_WD-WMC300004469      -      -    198    438  24.4M  51.7M
    ata-WDC_WD20EFRX-68EUZN0_WD-WCC4MLK57MVX      -      -    306    384  37.8M  45.1M
  mirror                                       651G  1.18T    510    457  63.2M  49.6M
    ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0429536      -      -    304    371  37.8M  43.3M
    ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M0VYKFCE      -      -    206    423  25.5M  49.6M

This is where I wonder what's going on

zpool iostat during read

                                                 capacity     operations    bandwidth
pool                                          alloc   free   read  write   read  write
--------------------------------------------  -----  -----  -----  -----  -----  -----
pool2                                         1.27T  2.35T  2.68K     32   339M   141K
  mirror                                       651G  1.18T  1.34K     20   169M  90.0K
    ata-WDC_WD20EFRX-68AX9N0_WD-WMC300004469      -      -    748      9  92.5M  96.8K
    ata-WDC_WD20EFRX-68EUZN0_WD-WCC4MLK57MVX      -      -    623     10  76.8M  96.8K
  mirror                                       651G  1.18T  1.34K     11   170M  50.8K
    ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0429536      -      -    774      5  95.7M  56.0K
    ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M0VYKFCE      -      -    599      6  74.0M  56.0K

iostat -x during the same read operation. Note how IO % is not at 100%.

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.60     0.00  661.30    6.00 83652.80    49.20   250.87     2.32    3.47    3.46    4.87   1.20  79.76
sdd               0.80     0.00  735.40    5.30 93273.20    49.20   251.98     2.60    3.51    3.51    4.15   1.20  89.04
sdf               0.50     0.00  656.70    3.80 83196.80    31.20   252.02     2.23    3.38    3.36    6.63   1.17  77.12
sda               0.70     0.00  738.30    3.30 93572.00    31.20   252.44     2.45    3.33    3.31    7.03   1.14  84.24

zpool and test dataset settings:

• atime is off
• compression is off
• ashift is 0 (autodetect - my understanding was that this was ok)
• zdb says disks are all ashift=12
• module - options zfs zvol_threads=32 zfs_arc_max=17179869184
• sync = standard

Edit - Oct, 30, 2015

I did some more testing

• dataset bonnie++ w/recordsize=1M = 226MB write, 392MB read much better
• dataset dd w/record size=1M = 260MB write, 392MB read much better
• zvol w/ext4 dd bs=1M = 128MB write, 107MB read why so slow?
• dataset 2 processess in parallel = 227MB write, 396MB read
• dd direct io makes no different on dataset and on zvol

I am much happier with the performance with the increased record size. Almost every file on the pool is way over 1MB. So I'll leave it like that. The disks are still not getting 100% utilization, which makes me wonder if it could still be much faster. And now I'm wondering why the zvol performance is so lousy, as that is something I (lightly) use.

I am happy to provide any information requested in the comments/answers. There is also tons of information posted in my other question: Slow copying between NFS/CIFS directories on same server

I am fully aware that I may just not understand something and that this may not be a problem at all. Thanks in advance.

To make it clear, the question is: Why isn't the ZFS pool as fast as I expect? And perhaps is there anything else wrong?

Please bear with me, I know it's a lot to read. This problem may be applicable to others, so it would be great to have an answer. I had to give away the bounty because it was going to expire.

When I copy to or from my NFS server (Debian) from a client (Ubuntu), it maxes out the gigabit. But, when I copy between two directories on the same server, it's speed bounces around between < 30MB/sec up to over 100MB/sec. Most of the time it's around 50MB/sec.

The same copy performed directly on the NFS server (local disks) I get 100-150 MB/sec, sometimes more. A file copy between this NFS export and a CIFS share exported from the same directory on the same server is just as slow and a copy between two directories over CIFS on the same server is slow. iperf shows bidirectional speed is 941Mb/940Mb between the client and server.

I made sure NFS is using async on the server. I also disabled sync on the ZFS dataset and tried removing the ZFS cache and log devices.

I've tested on a very fast ZFS striped mirror of 4x2TB disks, with an SSD for log and cache devices.

NFS server specs:

Debian 8.2 core 4Ghz AMD-FX
32GB ram
ZFS raid 10, SSD cache/log
17GB ARC
4x2GB WD red drives
Intel 82574L NIC

Test client:

Ubuntu 15.04, Core2Quad 2.4Ghz
8GB ram
SSD
Intel 82574L NIC

This is how things are currently set up. /pool2/Media is the share I've been testing with.

/etc/fstab on client:

UUID=575701cc-53b1-450c-9981-e1adeaa283f0 /               ext4        errors=remount-ro,discard,noatime,user_xattr 0       1
UUID=16e505ad-ab7d-4c92-b414-c6a90078c400 none            swap    sw              0       0 
/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0
tmpfs    /tmp    tmpfs   mode=1777       0       0


igor:/pool2/other     /other        nfs         soft,bg,nfsvers=4,intr,rsize=65536,wsize=65536,timeo=50,nolock
igor:/pool2/Media       /Media          nfs     soft,bg,nfsvers=4,intr,rsize=65536,wsize=65536,timeo=50,nolock,noac
igor:/pool2/home        /nfshome        nfs     soft,bg,nfsvers=4,intr,rsize=65536,wsize=65536,timeo=50,nolock

/etc/exports on server (igor):

#LAN
/pool2/home 192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash)
/pool2/other 192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash)
/pool2/Media 192.168.1.0/24(rw,async,no_subtree_check,no_root_squash)
/test 192.168.1.0/24(rw,async,no_subtree_check,no_root_squash)

#OpenVPN
/pool2/home 10.0.1.0/24(rw,sync,no_subtree_check,no_root_squash)
/pool2/other 10.0.1.0/24(rw,sync,no_subtree_check,no_root_squash)
/pool2/Media 10.0.1.0/24(rw,sync,no_subtree_check,no_root_squash)

zpool status:

  pool: pool2
 state: ONLINE
  scan: scrub repaired 0 in 6h10m with 0 errors on Sat Oct  3 08:10:26 2015
config:

        NAME                                                 STATE     READ WRITE CKSUM
        pool2                                                ONLINE       0     0     0
          mirror-0                                           ONLINE       0     0     0
            ata-WDC_WD20EFRX-68AX9N0_WD-WMC300004469         ONLINE       0     0     0
            ata-WDC_WD20EFRX-68EUZN0_WD-WCC4MLK57MVX         ONLINE       0     0     0
          mirror-1                                           ONLINE       0     0     0
            ata-WDC_WD20EFRX-68AX9N0_WD-WCC1T0429536         ONLINE       0     0     0
            ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M0VYKFCE         ONLINE       0     0     0
        logs
          ata-KINGSTON_SV300S37A120G_50026B7751153A9F-part1  ONLINE       0     0     0
        cache
          ata-KINGSTON_SV300S37A120G_50026B7751153A9F-part2  ONLINE       0     0     0

errors: No known data errors

  pool: pool3
 state: ONLINE
  scan: scrub repaired 0 in 3h13m with 0 errors on Sat Oct  3 05:13:33 2015
config:

        NAME                                        STATE     READ WRITE CKSUM
        pool3                                       ONLINE       0     0     0
          ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E5PSCNYV  ONLINE       0     0     0

errors: No known data errors

/pool2 bonnie++ on server:

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
igor            63G   100  99 187367  44 97357  24   325  99 274882  27 367.1  27

Bonding

I tried bonding and with a direct connection, balance-rr bonding, I get 220MB/sec read and 117MB/sec write, 40-50MB/sec copy.

iperf with bonding

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec  707             sender
[  4]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver
[  6]   0.00-10.00  sec  1.06 GBytes   909 Mbits/sec  672             sender
[  6]   0.00-10.00  sec  1.06 GBytes   908 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  2.15 GBytes  1.85 Gbits/sec  1379             sender
[SUM]   0.00-10.00  sec  2.15 GBytes  1.85 Gbits/sec                  receiver

Bonnie++ with bonding over NFS

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
haze            16G  1442  99 192941  16 89157  15  3375  96 179716  13  6082  77

With the ssd cache/log removed, copying over NFS, iostat shows this

sdb               0.80     0.00   67.60  214.00  8561.60 23689.60   229.06     1.36    4.80   14.77    1.64   1.90  53.60
sdd               0.80     0.00   54.60  214.20  7016.00 23689.60   228.46     1.37    5.14   17.41    2.01   2.15  57.76
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sde               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               1.60     0.00  133.00  385.20 17011.20 45104.00   239.73     2.24    4.31   12.29    1.56   1.57  81.60
sdf               0.40     0.00  121.40  385.40 15387.20 45104.00   238.72     2.36    4.63   14.29    1.58   1.62  82.16
sdg               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdh               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

TMPFS

I exported a tmpfs over NFS and did a file copy - The speed was 108MB/sec. Local from the server, it is 410MB/sec.

zvol mounted over NFS

The speed bounces around between < 50MB/sec up to > 180MB/sec, but averages out to about 100MB/sec. This is about what I'm looking for. This zvol is on the same pool (pool2) as I've been testing on. This really makes me think this is more of a ZFS dataset/caching type issue.

Raw disk read test

Using this command

dd if=/dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WMC300004469 of=/dev/null bs=1M count=2000

I get 146-148MB/sec for all 4 disks

Slow, uneven disk usage in pool

Thanks to a very helpful person on the ZFS mailing list, I know what to do to get more even usage of the disks.

The reason for ZFS to prefer mirror-1 is that it seems to be added after mirror-0 had been filled quite a bit, now ZFS is trying to rebalance the fill level.

In case you want to get rid of that and have some time: Iteratively zfs send the datasets of the pool to new datasets on itself, then destroy the source, repeat until pool is rebalanced.

I've fixed this, the data is level across all disks now This has resulted in a 75MB/sec copy speed over NFS. And 118MB/sec local.

The question

My question(s). If you can answer any one of the questions I will accept your answer:

1. How can my problem be solved? (slow copy over NFS, but not local)
2. If you can't answer #1, can you try this on your comparable NFS server with ZFS on Linux and tell me the results so I have something to compare it to?
3. If you can't answer #1 or #2, can you try the same testing on a similar but non-ZFS server over NFS?