Michael's questions

I setup the ZFS-HA following the excellent description Github (see here). After extensive testing, I rolled the setup out to production using 5x12 disks in RAIDZ3 connected to two nodes using HBA Controllers. This ran quite smooth until last night when one of the two storage pools suddenly faulted with "The pool metadata is corrupted." during a scrub run. At this point I can only speculate about what caused this, both pools were set up with SCSI fencing in pacemaker and disk reservations worked flawlessly during all failure scenarios I tested before going into production. The only major incident which occurred recently were two complete power outages without UPS support (read: the power was just gone from one moment to the next). However, it might also be that the true reason for the corruption is something completely different.

The situation now is that I cannot import the pool anymore (kindly see the output of zpool import at the end of this question). So far, all my intents to rescue the pool failed:

# zpool import -f tank
cannot import 'tank': one or more devices is currently unavailable

# zpool import -F tank
cannot import 'tank': one or more devices is currently unavailable

This puzzles me a bit since it does not really say that the only option would be to destroy the pool (which would be the expected response on a lethally corrupted pool).

# zpool clear -F tank
cannot open 'tank': no such pool

I also manually removed all SCSI reservations, e.g.:

# DEVICE=35000c5008472696f
# sg_persist --in --no-inquiry --read-reservation --device=/dev/mapper/$DEVICE
# sg_persist --in --no-inquiry --read-key --device=/dev/mapper/$DEVICE
# sg_persist --out --no-inquiry --register --param-sark=0x80d0001 --device=/dev/mapper/$DEVICE
# sg_persist --out --no-inquiry --clear --param-rk=0x80d0001 --device=/dev/mapper/$DEVICE
# sg_persist --in --no-inquiry --read-reservation --device=/dev/mapper/$DEVICE

I further tried removing A/C from the disk shelves to clear any temporary information that might remain in the desks.

I am quite frankly running short on options. The only thing left on my list is the -X option to zpool import - which I will try after all other measures failed.

So my question is, did you run into anything like this before and - more importantly - did you find a way to resolve this? I would be very grateful for any suggestions you might have .

=========

Pool layout/configuration:

   pool: tank
     id: 1858269358818362832
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:

        tank                   FAULTED  corrupted data
          raidz3-0             FAULTED  corrupted data
            35000c5008472696f  ONLINE
            35000c5008472765f  ONLINE
            35000c500986607bf  ONLINE
            35000c5008472687f  ONLINE
            35000c500847272ef  ONLINE
            35000c50084727ce7  ONLINE
            35000c50084729723  ONLINE
            35000c500847298cf  ONLINE
            35000c50084728f6b  ONLINE
            35000c50084726753  ONLINE
            35000c50085dd15bb  ONLINE
            35000c50084726e87  ONLINE
          raidz3-1             FAULTED  corrupted data
            35000c50084a8a163  ONLINE
            35000c50084e80807  ONLINE
            35000c5008472940f  ONLINE
            35000c50084a8f373  ONLINE
            35000c500847266a3  ONLINE
            35000c50084726307  ONLINE
            35000c50084726897  ONLINE
            35000c5008472908f  ONLINE
            35000c50084727083  ONLINE
            35000c50084727c8b  ONLINE
            35000c500847284e3  ONLINE
            35000c5008472670b  ONLINE
          raidz3-2             FAULTED  corrupted data
            35000c50084a884eb  ONLINE
            35000c500847262bb  ONLINE
            35000c50084eb9f43  ONLINE
            35000c50085030a4b  ONLINE
            35000c50084eb238f  ONLINE
            35000c50084eb6873  ONLINE
            35000c50084728baf  ONLINE
            35000c50084eb4c83  ONLINE
            35000c50084727443  ONLINE
            35000c50084a8405b  ONLINE
            35000c5008472868f  ONLINE
            35000c50084727c6f  ONLINE
          raidz3-3             FAULTED  corrupted data
            35000c50084eaa467  ONLINE
            35000c50084e7d99b  ONLINE
            35000c50084eb55e3  ONLINE
            35000c500847271d7  ONLINE
            35000c50084726cef  ONLINE
            35000c50084726763  ONLINE
            35000c50084727713  ONLINE
            35000c50084728127  ONLINE
            35000c50084ed0457  ONLINE
            35000c50084e5eefb  ONLINE
            35000c50084ecae2f  ONLINE
            35000c50085522177  ONLINE
          raidz3-4             FAULTED  corrupted data
            35000c500855223c7  ONLINE
            35000c50085521a07  ONLINE
            35000c50085595dff  ONLINE
            35000c500855948a3  ONLINE
            35000c50084f98757  ONLINE
            35000c50084f981eb  ONLINE
            35000c50084f8b0d7  ONLINE
            35000c50084f8d7f7  ONLINE
            35000c5008539d9a7  ONLINE
            35000c5008552148b  ONLINE
            35000c50085521457  ONLINE
            35000c500855212b3  ONLINE

Edit:

Servers are 2x Dell PowerEdge R630, Controllers are DELL OEM versions of Broardcom SAS HBA (should be similar to SAS 9300-8e) and all 60 disks in this pool are Seagate ST6000NM0034. The Enclosure is Quanta MESOS M4600H.

Edit 2:

OS is CentOS 7

ZFS is zfs-0.7.3-1.el7_4.x86_64

I have the following setup:

A single server with two LSI MegaRAID SAS 9380-8e controllers which are both connected to two 60-bay disk shelves while roughly following the design by Edmund White (see https://github.com/ewwhite/zfs-ha/wiki). The goal is to replicate the exact setup, but it's currently mid-migration.

After wiring the first shelf, all 60 disks were seen by both controllers and multipathing was setup and works smoothly. When adding the second disk shelf, there was still some old RAID configuration on the 60 disks which was dutifully reported by both controllers. Using the first controller I removed the configuration from disks and set them to being JBOD. All 60 disks are now visible to the OS and could be registered with multipath but only report a single path (going through controller 1), the second controller still reports all 60 disks as foreign (UGood F) and there is seemingly no way to forcibly make the controller rescan the devices or forget the current config for just this shelf:

# /opt/MegaRAID/storcli/storcli64 /c1 /e71 /sall show | head -n20
Controller = 1
Status = Success
Description = Show Drive Information Succeeded.


Drive Information :
=================

-----------------------------------------------------------------------
EID:Slt DID State DG     Size Intf Med SED PI SeSz Model            Sp 
-----------------------------------------------------------------------
71:0     74 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  
71:1    107 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  
71:2     72 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  
71:3     95 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  
71:4     90 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  
71:5     77 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  
71:6     73 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  
71:7     76 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  
71:8     83 UGood F  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  D  

This is the same shelf as seen by the other controller:

# /opt/MegaRAID/storcli/storcli64 /c0 /e165 /sall show | head -n20
Controller = 0
Status = Success
Description = Show Drive Information Succeeded.


Drive Information :
=================

-----------------------------------------------------------------------
EID:Slt DID State DG     Size Intf Med SED PI SeSz Model            Sp 
-----------------------------------------------------------------------
165:0   127 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  
165:1   121 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  
165:2   118 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  
165:3   116 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  
165:4   146 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  
165:5   122 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  
165:6   115 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  
165:7   142 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  
165:8   145 JBOD  -  3.637 TB SAS  HDD N   N  512B HUS724040ALS640  U  

But trying to clear the (wrong) info from the second controller does not work:

# /opt/MegaRAID/storcli/storcli64 /c1 /fall show
Controller = 1
Status = Success
Description = Couldn't find any foreign Configuration

# /opt/MegaRAID/storcli/storcli64 /c1 /fall delete
Controller = 1
Status = Success
Description = Couldn't find any foreign Configuration

# /opt/MegaRAID/storcli/storcli64 /c1 /fall import
Controller = 1
Status = Success
Description = Couldn't find any foreign Configuration

Forcing the disks into JBOD on the second controller does not work either:

# /opt/MegaRAID/storcli/storcli64 /c1 /e71 /sall set jbod | head -n20
Controller = 1
Status = Failure
Description = Set Drive JBOD Failed.

Detailed Status :
===============

-------------------------------------------------
Drive       Status  ErrCd ErrMsg                 
-------------------------------------------------
/c1/e71/s0  Failure   255 Operation not allowed. 
/c1/e71/s1  Failure   255 Operation not allowed. 
/c1/e71/s2  Failure   255 Operation not allowed. 
/c1/e71/s3  Failure   255 Operation not allowed. 
/c1/e71/s4  Failure   255 Operation not allowed. 
/c1/e71/s5  Failure   255 Operation not allowed. 
/c1/e71/s6  Failure   255 Operation not allowed. 
/c1/e71/s7  Failure   255 Operation not allowed. 
/c1/e71/s8  Failure   255 Operation not allowed. 
/c1/e71/s9  Failure   255 Operation not allowed. 

Is there any way to tell the RAID controller those disks do no longer have a foreign config and should be seen as JBODs?

I am in the process of setting up a ZFS storage system with a large number of disks. I want to use WWN numbers to identify devices (aliased in vdev_id.conf) but when trying to get the WWN numbers, I noted a shift between the WWN reported by the LSI controller (LSI MegaRAID SAS 9380-8e, all disks are in JBOD mode) and the WWN shown in /dev/disk/by-id:

$ storcli64 /c0/e72/s0 show all | grep 5008472696
WWN = 5000C5008472696C
0 Active 6.0Gb/s   0x5000c5008472696d 
1 Active 6.0Gb/s   0x5000c5008472696e 

$ ll /dev/disk/by-id/wwn-0x* | grep 5008472696
/dev/disk/by-id/wwn-0x5000c5008472696f -> ../../sdn

On this set of hardware (Disks, JBOD, controller), the pattern seems to be consistent (shifted by 3).

I have a 'feeling' that this is caused by the dual link but I could not find a suitable explanation for this behavior. Any suggestions will be greatly appreciated.

EDIT:

Another sample would be

WWN = 5000C50084726B78
0 Active 6.0Gb/s   0x5000c50084726b79 
1 Active 6.0Gb/s   0x5000c50084726b7a 

The controller reports 0x5000c50084726b7b, which is consistent with the suggested explanation given by Matthew.

I have a disk shelve with 30x 6TB disks and a HW RAID Controller. I plan to set those up as RAID 10, giving me 90TB of usable space for write-heavy workloads.

The minimum number of disks for RAID 10 is 4 and while each additional pair of disks would still increase IOPS the actual gain decreases. I found that an (unofficial) optimum would be around 6 to 8 disks per RAID 10 volume (giving about 225-400 IOPS with 7.2k SATA drives).

Most example on the web consider 4-disk setups. I found (only) one resource implicitly suggesting to build multiple small RAID10 volumes and concatenate (not: stripe) those using LVM.

What would be the better (i.e. technically sound) setup:

• 5x 6-disk RAID10 concatenated by LVM
• 1x 30-disk RAID10 (if supported by the controller)
• any alternative solution I am missing

I have a 44-bay Supermicro JBOD which is connected to a MegaRAID 9280-8e. Upon boot, all drives show activity and the device scan reaches 100%. Then however, an eror message states:

Invalid SAS topology, please check your cabling, fix and restart.

(Or something highly similar, this is from memory).

The RAID controller has proven to work on a different JBOD and I was unable to find any option in the RAID or system BIOS related to SAS topology, so I assume it has to be some issue with the JBOD.

I did the following steps to identify the problem:

1. I pulled all disks to verify it's not an issue caused by a single faulty drive. The error message remains identical ("Invalid SAS topology")

2. I verified the internal SAS cabling is accodring the backplanes manuals (IN -> backplane1 -> backplane2 -> IN/OUT).

3. Leaving the system running for a while, it repeatedly prints the "Invalid SAS topology" error but eventually continues and allows to enter the RAID configuration tool. Once there I see all disks and can even assemble RAID6 volumes. The volumes are still there on next boot, as is the error message "Invalid SAS topology".

I am quite puzzled and would be very grateful for any help or suggestions.

I need to set up an XFS filesystem on top of LVM on top of a hardware RAID-6 (10x 6TB + 2 parity) and I found the guideline on http://xfs.org/index.php/XFS_FAQ ("How to calculate the correct sunit,swidth values for optimal performance") which recommends:

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

I understand that the value for sunit is the stripe size defined during RAID volume creation and swidth is the number of data disks (e.g. 10 in the example above).

If I want to create e.g. 2 logical volumes with 12 TB and 30 TB, would I still use swidth=10 for both XFS filesystems or would I use swidth=2 and swidth=5 consistent with the 2 and 5 data disks which (mathematically) make up those logical volumes?