Questions[checksum](server)

I'm running a vendor-specific derivative of FreeNAS 9.3.

My trouble started when I installed a new JBOD chassis to add two new vdevs into my pool, and the chassis had a bad board. During this time, I was seeing SAS power errors on the drives on the bad board--my new drives were effectively turning on and off again, repeatedly, every minute.

I replaced the board and now, by most measures, the drives are functioning well, but ZFS is still giving me extremely strange checksum errors when I view zpool status. I think there were some bad CoW writes when I was having the SAS power issues.

The first chassis with the CPU, boot drive, RAM, etc., connects to the first expansion JBOD chassis via mini-SAS, and the second JBOD expansion chassis is daisy-chained through the first JBOD expansion chassis, also via mini-SAS.

• [Chassis 1: boot drive, two L2ARC SSDs, 11/11 drives of RAIDZ3-0, 1/11 drives RAIDZ3-1] -->mini-SAS to Chassis 2
• [Chassis 2: 10/11 drives of RAID Z3-1, 6/11 drives of RAID Z3-2] -->mini-SAS to Chassis 3
• [Chassis 3: 5/11 drives of RAIDZ3-2, 11/11 drives of RAIDZ3-3]

The checksum errors don't neatly map to any one controller or chassis, but my hunch is that when I was having these power issues, whatever data was being written to the different new disks was being written badly across the two new vdevs.

My HBAs are on good LSI firmware--all are on 20.00.04.00 or 20.00.08.00

I've swapped mini-SAS cables, and tried using different ports, to no avail.

The output of zpool statusis showing checksum errors accumulating on the two new vdevs, and after either a scrub, reboot, or zpool clear, eventually zpool status marks those vdevs as degraded. What's strange is that it also marks some of the drives that belong to those vdevs as degraded, but their actual error count of the individual disks are all 0. zdb shows that the individual drives are marked degraded because they have too many checksum errors, even though all their checksum error counts are actually 0. What's also strange is that the pool-level checksum errors show a lower number than the checksum errors from the two problem vdevs added together.

zpool status -v persistently shows a permanent error in a snapshot mapped to a 0x0 inode that has long been deleted, but can't seem to be cleared by multiple scrubs, reboots, or zpool clear. Also, other permanent errors float in and out, sometimes only showing as hexcode inodes, and other times as part of recent snapshots. I can't find any 0x0 with lsof.

I believe that there might be some kind of data corruption with the metadata in the pool.

I'm looking for a way to surgically remove these phantom snapshots or otherwise return my pool to a healthy state without destroying my data. I suspect that somewhere, ZFS is iterating over these corrupt phantom snapshots and causing both the bizarre checksum errors and the degraded states on the vdevs.

I have "cold" LTO backups of much of my important data, but otherwise, if I can't repair my pool, I'm preparing to set up a second server, offload everything to the "hot" second server, destroy my pool at the top level, and then reload from the hot backup.

Here's the output of zpool status -v:

[root@Jupiter] ~# zpool status -v
  pool: freenas-boot
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool.
  scan: resilvered 944M in 0h17m with 0 errors on Tue Aug  9 11:56:28 2016
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        da46p2  ONLINE       0     0     0  block size: 8192B configured, 8388608B native
        da47p2  ONLINE       0     0     0  block size: 8192B configured, 8388608B native

errors: No known data errors

  pool: pool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub in progress since Fri Sep  9 22:43:51 2016
        6.27T scanned out of 145T at 1.11G/s, 35h27m to go
        0 repaired, 4.33% done
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool                                            DEGRADED     0     0   118
      raidz3-0                                      ONLINE       0     0     0
        gptid/ac108605-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ac591d4e-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ac92fd0d-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/accd3076-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ad067e97-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ad46cbee-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ad91ba17-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/adcbdd0a-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ae07dc0d-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ae494d10-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
        gptid/ae93a3a5-265c-11e5-9a02-0cc47a599098  ONLINE       0     0     0
      raidz3-1                                      ONLINE       0     0     0
        gptid/12f6a4c5-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/511ea1f9-1932-11e6-9b1e-0cc47a599098  ONLINE       0     0     0
        gptid/14436fcf-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/14f50aa3-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/159b5654-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/163d682b-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/16ee624e-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/1799dde3-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/184c2ea4-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/18f51c30-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
        gptid/19a861ea-c929-11e5-8075-0cc47a599098  ONLINE       0     0     0
      raidz3-2                                      DEGRADED     0     0   236
        gptid/5f80fc42-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/60369e0f-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/60e8234a-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/61a235f2-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/62580471-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/6316a38a-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/63d4bce8-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/ebfc2b99-6893-11e6-9b09-0cc47a599098  ONLINE       0     0     0
        gptid/654f143a-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/66236b33-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/66eda3f6-4e00-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
      raidz3-3                                      DEGRADED     0     0   176
        gptid/c77a9da9-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/c83e100e-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/c8fd9ced-4e02-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/c9bb21ba-4e02-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/ca7a48db-4e02-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/cb422329-4e02-11e6-b7cf-0cc47a599098  DEGRADED     0     0     0  too many errors
        gptid/cbfe4c21-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/ccc43528-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/cd93a34c-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/ce622f51-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
        gptid/cf2591d3-4e02-11e6-b7cf-0cc47a599098  ONLINE       0     0     0
    cache
      gptid/aedd3872-265c-11e5-9a02-0cc47a599098    ONLINE       0     0     0
      gptid/af559c10-265c-11e5-9a02-0cc47a599098    ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <0x357>:<0x2aef3>
        <0x37b>:<0x397285>
pool/[email protected]:<0x0>

Via the FreeNAS GUI, I tried copying the System dataset pool from pool over to freenas-boot and then tried using zfs destroy to delete the pool copy of pool/.system and leaving the freenas-boot copy intact. I was able to use zfs destroy to delete everything within pool/.system listed in zfs list, but upon trying to destroy pool/.system with zfs destroy, the shell returned the error: Cannot iterate filesystems: I/O error. I tried zfs destroy on pool/.system with the the -f, -r, and -R flags, as per the Oracle ZFS documentation, to no avail.

I started yet another scrub. Perhaps eliminating the contents of pool/.system on the pool copy of the System dataset pool will allow the scrub to clear out the metadata error with the phantom snapshot pool/[email protected].

I'm wondering if it's possible to resilver each disk that's showing up as degraded, one-by-one, so that the "bad" metadata that's not being reference can be abandoned. I've resilvered two disks, but now I'm running into an issue wherein resilvering any additional disk causes the other disks I've already resilvered to begin resilvering again at the same time. I believe it might be a ZFS bug related to periodic snapshot tasks, and I've gone ahead and deleted my periodic snapshot task and destroyed all my snapshots, but I'm hesitant to try to resilver yet another one of the degraded drives for fear that all the previously resilvered disks will resilver again, leaving me without any redundancy, eventually to the point of having a faulted pool.

After disabling my periodic snapshot tasks and deleting all my snapshots, I tried wiping one disk and then resilvering it, but the three disks that I'd already resilvered started resilvering again. Now I'm almost certain that I'd have two different disks per each problem RAID-Z3 vdev that would resilver, so if I attempt to resilver any more disks, I'll lose the redundancy in each of the problem vdevs and my pool will fault.

One other bizarre behavior is that checking zpool status -v actually increases the pool's checksum error count incrementally, but checking zpool status does not. It's almost as if the -v flag itself is iterating over whatever mechanism is causing checksum errors.

Would using zdb -c on my pool somehow be able to "fix" these metadata errors?

A large, USB-connected HDD with btrfs contains a 1TB file (a disk image). The first btrfs scrub after writing that file found 3 logical bad blocks developed somewhere in the middle of that file:

[ 7702.964265] BTRFS warning (device sdd1): checksum error at logical
5473719291904 on dev /dev/sdd1, sector 222940168, root 5, inode 1245769,
offset 97110921216, length 4096, links 1 (path: dysk/dysk.bin)
[ 7702.964274] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 17, gen 0
[ 7702.964278] BTRFS error (device sdd1): unable to fixup (regular) error
at logical 5473719291904 on dev /dev/sdd1
[…]
[ 9588.625906] BTRFS warning (device sdd1): checksum error at logical
5241172611072 on dev /dev/sdd1, sector 691494312, root 5, inode 1245769,
offset 310632271872, length 4096, links 1 (path: dysk/dysk.bin)
[ 9588.625916] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 18, gen 0
[ 9588.625920] BTRFS error (device sdd1): unable to fixup (regular) error
at logical 5241172611072 on dev /dev/sdd1
[…]
[20545.302898] BTRFS warning (device sdd1): checksum error at logical
3991747321856 on dev /dev/sdd1, sector 3185809200, root 5, inode 1245769,
offset 907925676032, length 4096, links 1 (path: dysk/dysk.bin)
[20545.302908] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 19, gen 0
[20545.302912] BTRFS error (device sdd1): unable to fixup (regular) error
at logical 3991747321856 on dev /dev/sdd1

No backups for this file, so no perfect recovery possible. The full area of the btrfs partition can be read without I/O error, so I think there are two possible cases: either the data block was written incorrectly (unrecoverable) or an incorrect checksum is stored (fix should be simple: recompute the checksum). Given that the first case is already lost, how can I ask btrfs to recompute the checksum of a data block?

So rsync runs some checksums in the course of deciding what to transfer (i.e. what blocks within a file). But is there any reason to trust the file you end up with on the receive side any more than you would for a normal network transfer? Should I run checksums after rsync finishes to verify the data? Is rerunning rsync with the pre-check (i.e. --checksum option) turned on an accepted way to accomplish this?

I find too many incorrect checksum errors from a TCPDUMP done on a GNU Linux 64bit server. There are close to 50% incorrect chekcsums in the export?

cksum 0xe61f (incorrect (-> 0x8c37)

How can we interpret this data? Does it affect performance a lot?

Here's my problem : I need to archive to tar files a lot ( up to 60 TB) of big files (usually 30 to 40 GB each). I would like to make checksums ( md5, sha1, whatever) of these files before archiving; however not reading every file twice (once for checksumming, twice for tar'ing) is more or less a necessity to achieve a very high archiving performance (LTO-4 wants 120 MB/s sustained, and the backup window is limited).

So I'd need some way to read a file, feeding a checksumming tool on one side, and building a tar to tape on the other side, something along :

tar cf - files | tee tarfile.tar | md5sum -

Except that I don't want the checksum of the whole archive (this sample shell code does just this) but a checksum for each individual file in the archive.

I've studied GNU tar, Pax, Star options. I've looked at the source from Archive::Tar. I see no obvious way to achieve this. It looks like I'll have to hand-build something in C or similar to achieve what I need. Perl/Python/etc simply won't cut it performance-wise, and the various tar programs miss the necessary "plugin architecture". Does anyone know of any existing solution to this before I start code-churning ?