ZFS on Linux Server occasionally stalls and then resumes

Im having an issue which I have a really hard time debugging. Running ZFS, by system "hiccuped", dumped some information into DMESG, and continued working.

My ZFS is hosting VMs on ESXi. When this issue occurs, many of the VMs experience block IO errors, and some of them drop into read-only mode, requiring restores from backup or fsck to repair the filesystems. This issue only occurs very occasionally, and I have hammered the system, trying to stress it out, it does not seem to be performance related. Only occurs every few months, so conclusively solving it seems to be a pipe-dream to me.

First off, some info about my system (Centos 7, 4.5).

[root@zfs-head ~]# name -a

Linux zfs-head 4.5.0-1.el7.elrepo.x86_64 #1 SMP Mon Mar 14 10:24:58 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

dmesg entries:

[4331253.022999] sd 2:0:28:0: [sdaa] tag#2 CDB: Read(10) 28 00 10 a8 3d b5 00 00 20 00
[4331253.023006] mpt3sas_cm0:   sas_address(0x5000c500837f31f2), phy(8)
[4331253.023008] mpt3sas_cm0:   enclosure_logical_id(0x50010c60004d41ff),slot(0)
[4331253.023010] mpt3sas_cm0:   enclosure level(0x0003), connector name(     )
[4331253.023013] mpt3sas_cm0:   handle(0x002d), ioc_status(scsi data underrun)(0x0045), smid(222)
[4331253.023016] mpt3sas_cm0:   request_len(131072), underflow(16384), resid(131072)
[4331253.023018] mpt3sas_cm0:   tag(0), transfer_count(0), sc->result(0x00000000)
[4331253.023020] mpt3sas_cm0:   scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[4331253.023023] mpt3sas_cm0:   [sense_key,asc,ascq]: [0x06,0x2a,0x01], count(96)
[4331253.023030] sd 2:0:28:0: Mode parameters changed
[4331266.475222] sd 2:0:29:0: [sdab] tag#29 CDB: Write(10) 2a 00 09 97 6e c1 00 00 02 00
[4331266.475229] mpt3sas_cm0:   sas_address(0x5000c500837f25c6), phy(9)
[4331266.475232] mpt3sas_cm0:   enclosure_logical_id(0x50010c60004d41ff),slot(1)
[4331266.475234] mpt3sas_cm0:   enclosure level(0x0003), connector name(     )
[4331266.475237] mpt3sas_cm0:   handle(0x002e), ioc_status(scsi data underrun)(0x0045), smid(139)
[4331266.475239] mpt3sas_cm0:   request_len(8192), underflow(1024), resid(8192)
[4331266.475241] mpt3sas_cm0:   tag(0), transfer_count(0), sc->result(0x00000000)
[4331266.475244] mpt3sas_cm0:   scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[4331266.475246] mpt3sas_cm0:   [sense_key,asc,ascq]: [0x06,0x2a,0x01], count(96)
[4331266.475252] sd 2:0:29:0: Mode parameters changed

pool status:

[root@zfs-head ~]# pool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

    NAME                                             STATE     READ WRITE CKSUM
    storage                                          ONLINE       0     0     0
      mirror-0                                       ONLINE       0     0     0
        s1d1                                         ONLINE       0     0     0
        s2d1                                         ONLINE       0     0     0
      mirror-1                                       ONLINE       0     0     0
        s3d1                                         ONLINE       0     0     0
        s4d1                                         ONLINE       0     0     0
      mirror-2                                       ONLINE       0     0     0
        s1d2                                         ONLINE       0     0     0
        s2d2                                         ONLINE       0     0     0
      mirror-3                                       ONLINE       0     0     0
        s3d2                                         ONLINE       0     0     0
        s4d2                                         ONLINE       0     0     0
      mirror-4                                       ONLINE       0     0     0
        s1d3                                         ONLINE       0     0     0
        s2d3                                         ONLINE       0     0     0
      mirror-5                                       ONLINE       0     0     0
        s3d3                                         ONLINE       0     0     0
        s4d3                                         ONLINE       0     0     0
    logs
      ata-Samsung_SSD_850_PRO_128GB_S24ZNXAGA10768M  ONLINE       0     0     0
    cache
      ata-Samsung_SSD_850_EVO_250GB_S21NNXAG918721R  ONLINE       0     0     0
      ata-Samsung_SSD_850_EVO_250GB_S21NNXAGA59337A  ONLINE       0     0     0
      ata-Samsung_SSD_850_EVO_250GB_S21NNXAGA69590F  ONLINE       0     0     0

errors: No known data errors
[root@zfs-head ~]# 

My Vdev map:

[root@zfs-head ~]# cat /etc/zfs/vdev_id.conf
#     by-vdev
#     name     fully qualified or base name of device link
alias s1d1       /dev/disk/by-id/scsi-35000c500837ff247
alias s1d2       /dev/disk/by-id/scsi-35000c500837f15c3
alias s1d3       /dev/disk/by-id/scsi-35000c500837f137f
alias s2d1       /dev/disk/by-id/scsi-35000c500837f377b
alias s2d2       /dev/disk/by-id/scsi-35000c500837f5bf7
alias s2d3       /dev/disk/by-id/scsi-35000c500837f75bf
alias s3d1       /dev/disk/by-id/scsi-35000c500837f14d3
alias s3d2       /dev/disk/by-id/scsi-35000c500837f571b
alias s3d3       /dev/disk/by-id/scsi-35000c500837f604f
alias s4d1       /dev/disk/by-id/scsi-35000c500837f31f3
alias s4d2       /dev/disk/by-id/scsi-35000c500837f25c7
alias s4d3       /dev/disk/by-id/scsi-35000c500837f14cf

[root@zfs-head ~]# 

The box didn't restart, or really even acknowledge that there was an issue, save for the dmesg entries. I have googled those entries to my level best, but did not find anything relevant.

Help appreciated!