user260467's questions

Had to destroy and recreate a pool due to some other issues. I noticed that performance on the new pool is atrocious.

Whereas before on the old pool, ARC was full and ARC hits were high, ARC hits on the new pool have dropped considerably.

On the new pool, ARC just does not fill up.

The pattern of ARC hits are also strange. Having one or two clients try to play video, ARC would hit for a few seconds, and then plummet to 0, which causes the video playback to stutter and stop.

Why is performance so bad on the new pool?

I have two enterprise-grade SSDs that I've been using for L2ARC in an OpenZFS pool for a few years. Workflow during this time has been fairly heavy, with lots of reads and writes, serving up video to clients for video post-production.

Recently, got a couple of critical alerts about the L2ARC SSDs:

CRITICAL: Oct. 22, 2018, 6:12 p.m. - Device: /dev/da0 [SAT], 215869351264256 Offline uncorrectable sectors
CRITICAL: Oct. 22, 2018, 6:12 p.m. - Device: /dev/da1 [SAT], 174264003067904 Offline uncorrectable sectors

Obviously there's no risk of data loss, since these are just L2ARC, but is it time to just get them replaced?

Am I sacrificing performance? If not, is there a way to reset or ignore the bad sectors?

Using FreeNAS-11.1-U6.

When large read and write operations are conducted on a NFSv4 share on a CentOS 7.3 client, I notice that the NFS mounts often time out.

Does a fresh, out-of-the-box installation of CentOS 7.3 require letting the NFS service through the default firewall? If so, how?

I'm trying to use setfacl to set the permissions of different ZFS datasets, and I want those ACL permissions equivalent to UNIX-style chmod 770.

What setfacl command could accomplish this, and what ACLs should getfacl display if I were to do this correctly?

I'm running FreeNAS 11.0 with Samba 4.6.4.

I'm running a vendor-specific derivative of FreeNAS 9.3.

My trouble started when I installed a new JBOD chassis to add two new vdevs into my pool, and the chassis had a bad board. During this time, I was seeing SAS power errors on the drives on the bad board--my new drives were effectively turning on and off again, repeatedly, every minute.

I replaced the board and now, by most measures, the drives are functioning well, but ZFS is still giving me extremely strange checksum errors when I view zpool status. I think there were some bad CoW writes when I was having the SAS power issues.

The first chassis with the CPU, boot drive, RAM, etc., connects to the first expansion JBOD chassis via mini-SAS, and the second JBOD expansion chassis is daisy-chained through the first JBOD expansion chassis, also via mini-SAS.

• [Chassis 1: boot drive, two L2ARC SSDs, 11/11 drives of RAIDZ3-0, 1/11 drives RAIDZ3-1] -->mini-SAS to Chassis 2
• [Chassis 2: 10/11 drives of RAID Z3-1, 6/11 drives of RAID Z3-2] -->mini-SAS to Chassis 3
• [Chassis 3: 5/11 drives of RAIDZ3-2, 11/11 drives of RAIDZ3-3]

The checksum errors don't neatly map to any one controller or chassis, but my hunch is that when I was having these power issues, whatever data was being written to the different new disks was being written badly across the two new vdevs.

My HBAs are on good LSI firmware--all are on 20.00.04.00 or 20.00.08.00

I've swapped mini-SAS cables, and tried using different ports, to no avail.

The output of zpool statusis showing checksum errors accumulating on the two new vdevs, and after either a scrub, reboot, or zpool clear, eventually zpool status marks those vdevs as degraded. What's strange is that it also marks some of the drives that belong to those vdevs as degraded, but their actual error count of the individual disks are all 0. zdb shows that the individual drives are marked degraded because they have too many checksum errors, even though all their checksum error counts are actually 0. What's also strange is that the pool-level checksum errors show a lower number than the checksum errors from the two problem vdevs added together.

zpool status -v persistently shows a permanent error in a snapshot mapped to a 0x0 inode that has long been deleted, but can't seem to be cleared by multiple scrubs, reboots, or zpool clear. Also, other permanent errors float in and out, sometimes only showing as hexcode inodes, and other times as part of recent snapshots. I can't find any 0x0 with lsof.

I believe that there might be some kind of data corruption with the metadata in the pool.

I'm looking for a way to surgically remove these phantom snapshots or otherwise return my pool to a healthy state without destroying my data. I suspect that somewhere, ZFS is iterating over these corrupt phantom snapshots and causing both the bizarre checksum errors and the degraded states on the vdevs.

I have "cold" LTO backups of much of my important data, but otherwise, if I can't repair my pool, I'm preparing to set up a second server, offload everything to the "hot" second server, destroy my pool at the top level, and then reload from the hot backup.

Here's the output of zpool status -v:

[root@Jupiter] ~# zpool status -v
  pool: freenas-boot
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool.
  scan: resilvered 944M in 0h17m with 0 errors on Tue Aug  9 11:56:28 2016
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        da46p2  ONLINE       0     0     0  block size: 8192B configured, 8388608B native
        da47p2  ONLINE       0     0     0  block size: 8192B configured, 8388608B native

errors: No known data errors

  pool: pool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub in progress since Fri Sep  9 22:43:51 2016
        6.27T scanned out of 145T at 1.11G/s, 35h27m to go
        0 repaired, 4.33% done
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool                                            DEGRADED     0     0   118
      raidz3-0                                      ONLINE       0     0     0
        gptid/ac108605-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ac591d4e-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ac92fd0d-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/accd3076-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ad067e97-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ad46cbee-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ad91ba17-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/adcbdd0a-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ae07dc0d-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ae494d10-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ae93a3a5-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
      raidz3-1                                      ONLINE       0     0     0
        gptid/12f6a4c5-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/511ea1f9-1932-11e6-9b1e-0cc47a599098  ONLINE       0     0     0
        gptid/14436fcf-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/14f50aa3-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/159b5654-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/163d682b-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/16ee624e-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/1799dde3-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/184c2ea4-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/18f51c30-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/19a861ea-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
      raidz3-2                                      DEGRADED     0     0   236
        gptid/5f80fc42-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/60369e0f-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/60e8234a-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/61a235f2-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/62580471-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/6316a38a-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/63d4bce8-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/ebfc2b99-6893-11e6-9b09-0cc47a599098  ONLINE       0     0     0
        gptid/654f143a-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/66236b33-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/66eda3f6-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
      raidz3-3                                      DEGRADED     0     0   176
        gptid/c77a9da9-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/c83e100e-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/c8fd9ced-4e02-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/c9bb21ba-4e02-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/ca7a48db-4e02-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/cb422329-4e02-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/cbfe4c21-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/ccc43528-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/cd93a34c-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/ce622f51-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/cf2591d3-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
    cache
      gptid/aedd3872-265c-11e5-9a02-0cc47a599098    ONLINE       0     0     0
      gptid/af559c10-265c-11e5-9a02-0cc47a599098    ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <0x357>:<0x2aef3>
        <0x37b>:<0x397285>
pool/[email protected]:<0x0>

Via the FreeNAS GUI, I tried copying the System dataset pool from pool over to freenas-boot and then tried using zfs destroy to delete the pool copy of pool/.system and leaving the freenas-boot copy intact. I was able to use zfs destroy to delete everything within pool/.system listed in zfs list, but upon trying to destroy pool/.system with zfs destroy, the shell returned the error: Cannot iterate filesystems: I/O error. I tried zfs destroy on pool/.system with the the -f, -r, and -R flags, as per the Oracle ZFS documentation, to no avail.

I started yet another scrub. Perhaps eliminating the contents of pool/.system on the pool copy of the System dataset pool will allow the scrub to clear out the metadata error with the phantom snapshot pool/[email protected].

I'm wondering if it's possible to resilver each disk that's showing up as degraded, one-by-one, so that the "bad" metadata that's not being reference can be abandoned. I've resilvered two disks, but now I'm running into an issue wherein resilvering any additional disk causes the other disks I've already resilvered to begin resilvering again at the same time. I believe it might be a ZFS bug related to periodic snapshot tasks, and I've gone ahead and deleted my periodic snapshot task and destroyed all my snapshots, but I'm hesitant to try to resilver yet another one of the degraded drives for fear that all the previously resilvered disks will resilver again, leaving me without any redundancy, eventually to the point of having a faulted pool.

After disabling my periodic snapshot tasks and deleting all my snapshots, I tried wiping one disk and then resilvering it, but the three disks that I'd already resilvered started resilvering again. Now I'm almost certain that I'd have two different disks per each problem RAID-Z3 vdev that would resilver, so if I attempt to resilver any more disks, I'll lose the redundancy in each of the problem vdevs and my pool will fault.

One other bizarre behavior is that checking zpool status -v actually increases the pool's checksum error count incrementally, but checking zpool status does not. It's almost as if the -v flag itself is iterating over whatever mechanism is causing checksum errors.

Would using zdb -c on my pool somehow be able to "fix" these metadata errors?

I have an OS X client that is able to mount to SMB shares, but then arbitrarily disconnects. Sometimes the shares can stay mounted for hours, sometimes only 15 seconds. There doesn't seem to be any particular process that's causing the disconnections.

• Client is Mac OS X 10.11.6 (15G1004) client on a MacPro5,1.
• Client's NIC is an ATTO FastFrame NT12, with driver ATTOFastFrame.kext installed [v. 3.0.7]
• Client signing is disabled on the client as per Apple's slow SMB signing: https://discussions.apple.com/message/30287193#message30287193
• Server is a downstream derivative of FreeNAS 9.3 running Samba 4.3.6-GIT-UNKNOWN
• Shares are mounted as SMB3_02
• 10 GbE switch is NETGEAR ProSafe M7100-24x

Other OS X clients have no such issues on the same network and the same server with a Sonnet Twin 10G.

Other possibly relevant details:

• Boot drive is an OWC Mercury Accelsior with the driver properly installed, so there should be no power issues: https://eshop.macsales.com/tech_center/index.cfm?page=/manuals/item/owcaccelsior.html
• GPU is a 2014 NVIDIA GeForce GTX TITAN X with the MacVidCards firmware
• The disconnects are noticed while using Adobe Premiere Pro and Blackmagic DaVinci Resolve

Why might these disconnects be occurring in this way?