user124381's questions

I've started running out of space on one of my data pool and decided to expand it by adding some extra disks. Below is what it looked like before extending:

        NAME                       STATE     READ WRITE CKSUM
    data                       ONLINE       0     0     0
      raidz1-0                 ONLINE       0     0     0
        c0t5000C50041BF93A7d0  ONLINE       0     0     0
        c0t5000C50041FEBCFFd0  ONLINE       0     0     0
        c0t5000C50041FEBE1Bd0  ONLINE       0     0     0
        c0t5000C50041FEC223d0  ONLINE       0     0     0
        c0t5000C50041FECC0Bd0  ONLINE       0     0     0
        c0t5000C500422BAF33d0  ONLINE       0     0     0

Now after extending the pool I've got first set of drives in raidz1 and the second set using raidz2

        NAME                       STATE     READ WRITE CKSUM
    data                       ONLINE       0     0     0
      raidz1-0                 ONLINE       0     0     0
        c0t5000C50041BF93A7d0  ONLINE       0     0     0
        c0t5000C50041FEBCFFd0  ONLINE       0     0     0
        c0t5000C50041FEBE1Bd0  ONLINE       0     0     0
        c0t5000C50041FEC223d0  ONLINE       0     0     0
        c0t5000C50041FECC0Bd0  ONLINE       0     0     0
        c0t5000C500422BAF33d0  ONLINE       0     0     0
      raidz2-1                 ONLINE       0     0     0
        c0t5000C50034245B57d0  ONLINE       0     0     0
        c0t5000C500350A3847d0  ONLINE       0     0     0
        c0t5000C50042033AA7d0  ONLINE       0     0     0
        c0t5000C500421FBD03d0  ONLINE       0     0     0
        c0t5000C500422B9737d0  ONLINE       0     0     0
        c0t5000C5005678B8BFd0  ONLINE       0     0     0

This is my mistake, no excuse. My question is how do I undo this operation so I can setup the drives again to have raidz1-0 and raidz1-1? Data is being written to the data pool only during the night as this is a backup pool.

I'm having problem with few linux boxes running xen. They are acting as hypervisors and they are connected to SAN using multipath setup to provide storage to guest vms.

Every now and then one of two paths fails but it can be quickly restored by running:

multipath
multipath -ll

I need to get to the bottom of the issue and find out why this is happening. I have noticed that this doesn't occur when the hypervisor is not too busy (network and I/O wise). I have also eliminated possible hardware problem by moving all the services on to identical new chassis. I have collected few system logs which may indicate NIC module issue or kernel problem and failing multipath might be only a result of this!!?? Here is a bit of log which always shows up when multipath goes down:

kernel: BUG: soft lockup - CPU#0 stuck for 60s! [swapper:0]
kernel: BUG: soft lockup - CPU#2 stuck for 60s! [events/2:76]

I'll paste full logs at the end of this post to keep it easy to read. Now a little bit more about my setup:

• Internet access is setup over eth0 and eth2 (bonded)
• SAN multipath access is setup over eth1 and eth3

Server:

• Supermicro SuperServer 6016T-NTRF
• Intel(R) Xeon(R) CPU E5645
• Intel Corporation 82576 Gigabit Network

• CentOS release 5.7 (Final) 2.6.18-274.18.1.el5xen

• filename: /lib/modules/2.6.18-274.18.1.el5xen/kernel/drivers/net/igb/igb.ko

• version: 3.0.6-k2-1

• Log 01

• Log 02

If anyone needs more details please get it touch. Any help will be much appreciated.