Questions[multipath](server)

I am trying to copy(~7TB of data using rsync) between two server in same data center in the backend its using EMC VMAX3

After copying ~30-40GB of data multipath start failing

Dec 15 01:57:53 test.example.com multipathd: 360000970000196801239533037303434: Recovered to normal mode
Dec 15 01:57:53 test.example.com multipathd: 360000970000196801239533037303434: remaining active paths: 1
Dec 15 01:57:53 test.example.com kernel: sd 1:0:2:20: [sdeu]  Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK 

[root@test log]# multipath -ll |grep -i fail
 |- 1:0:0:15 sdq  65:0   failed ready running
  - 3:0:0:15 sdai 66:32  failed ready running

We are using default multipath.conf

HBA driver version  8.07.00.26.06.8-k

HBA model QLogic Corp. ISP8324-based 16Gb Fibre Channel to PCI Express Adapter

OS: CentOS 64-bit/2.6.32-642.6.2.el6.x86_64
Hardware:Intel/HP ProLiant DL380 Gen9

Already verified this solution and checked with EMC everything looks good https://access.redhat.com/solutions/438403

Some more info

- There is no drop/error packet on the network side.

• Filesystem is mounted with noatime,nodiratime
• Filesystem ext4(Already tried xfs but same error)
• LVM is in striped mode(Started with linear option and then converted to striped)

• Already disabled THP

• echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

• Whenever multipath start failing process goes to D state
• System firmware upgraded
• Tried with latest version of qlogic driver
• Tried with different scheduler(noop,deadline,cfq)
• Tried with different tuned profile(enterprise-storage)

Vmcore collected during the time of issue

I am able to collect vmcore during the time of issue

  KERNEL: /usr/lib/debug/lib/modules/2.6.32-642.6.2.el6.x86_64/vmlinux
DUMPFILE: vmcore  [PARTIAL DUMP]
    CPUS: 36
    DATE: Fri Dec 16 00:11:26 2016
  UPTIME: 01:48:57
  LOAD AVERAGE: 0.41, 0.49, 0.60
   TASKS: 1238
NODENAME: test.example.com
 RELEASE: 2.6.32-642.6.2.el6.x86_64
 VERSION: #1 SMP Wed Oct 26 06:52:09 UTC 2016
 MACHINE: x86_64  (2297 Mhz)
  MEMORY: 511.9 GB
   PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000018"
     PID: 15840
 COMMAND: "kjournald"
    TASK: ffff884023446ab0  [THREAD_INFO: ffff88103def4000]
     CPU: 2
   STATE: TASK_RUNNING (PANIC)

After Enbaling Debug mode on the qlogic sid

qla2xxx [0000:0b:00.0]-3822:5: FCP command status: 0x2-0x0 (0x70000) nexus=5:1:0 portid=1f0160 oxid=0x800 cdb=2a200996238000038000 len=0x70000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff882189d42580 cp=ffff88276d249480.
qla2xxx [0000:84:00.0]-3822:7: FCP command status: 0x2-0x0 (0x70000) nexus=7:0:3 portid=450000 oxid=0x4de cdb=2a20098a5b0000010000 len=0x20000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff882189d421c0 cp=ffff8880237e0880.

Unfortunately I may have to power-cycle my storage array.

In the context of using KVM with libvirt managing the Virtual Machines where i need to power-cycle the storage array, is it possible or even advisable to :

1. pause the VM's
2. flush and turn off multipath
3. log out of iSCSI sessions
4. update and power-cycle storage array.

then follow the reverse course of

1. re-establish iISCSI sessions
2. restore multipathing
3. resume VM's

Any advice is welcome.

Is it possible to set up bonding(active/backup) on top of existing vlan interfaces ?

Assume the following scenario:

• Linux Server with 2 NICs connected to separate switches
• iSCSI initiator + MPIO to utilize both NICs (redundancy+throughput)
  • each path via separate VLAN (e.g. VLAN 101+102)
• now I need another VLAN (e.g. 100) connected to the Server that should utilize NIC1 but failover to the other one if NIC1 looses it's link

Something like this:

    eth0  --- eth0.101 -- iSCSI IP 1
            \ eth0.100 ------------------- 
                                          \ __ bond100 -- IP
    eth1  --- eth0.102 -- iSCSI IP 2      /
            \ eth1.100 -------------------

I'm afraid this doesnt work, because the bonding driver doesn't get link-layer information from the vlan-interface...

Any other ideas how to solve this ?

I'm trying to optimize a storage setup on some Sun hardware with Linux. Any thoughts would be greatly appreciated.

We have the following hardware:

• Sun Blade X6270
• 2* LSISAS1068E SAS controllers
• 2* Sun J4400 JBODs with 1 TB disks (24 disks per JBOD)
• Fedora Core 12
• 2.6.33 release kernel from FC13 (also tried with latest 2.6.31 kernel from FC12, same results)

Here's the datasheet for the SAS hardware:

http://www.sun.com/storage/storage_networking/hba/sas/PCIe.pdf

It's using PCI Express 1.0a, 8x lanes. With a bandwidth of 250 MB/sec per lane, we should be able to do 2000 MB/sec per SAS controller.

Each controller can do 3 Gb/sec per port and has two 4 port PHYs. We connect both PHYs from a controller to a JBOD. So between the JBOD and the controller we have 2 PHYs * 4 SAS ports * 3 Gb/sec = 24 Gb/sec of bandwidth, which is more than the PCI Express bandwidth.

With write caching enabled and when doing big writes, each disk can sustain about 80 MB/sec (near the start of the disk). With 24 disks, that means we should be able to do 1920 MB/sec per JBOD.

multipath {
  rr_min_io 100
  uid 0
  path_grouping_policy multibus
  failback manual
  path_selector "round-robin 0"
  rr_weight priorities
  alias somealias
  no_path_retry queue
  mode 0644
  gid 0
  wwid somewwid
}

I tried values of 50, 100, 1000 for rr_min_io, but it doesn't seem to make much difference.

Along with varying rr_min_io I tried adding some delay between starting the dd's to prevent all of them writing over the same PHY at the same time, but this didn't make any difference, so I think the I/O's are getting properly spread out.

According to /proc/interrupts, the SAS controllers are using a "IR-IO-APIC-fasteoi" interrupt scheme. For some reason only core #0 in the machine is handling these interrupts. I can improve performance slightly by assigning a separate core to handle the interrupts for each SAS controller:

echo 2 > /proc/irq/24/smp_affinity
echo 4 > /proc/irq/26/smp_affinity

Using dd to write to the disk generates "Function call interrupts" (no idea what these are), which are handled by core #4, so I keep other processes off this core too.

I run 48 dd's (one for each disk), assigning them to cores not dealing with interrupts like so:

taskset -c somecore dd if=/dev/zero of=/dev/mapper/mpathx oflag=direct bs=128M

oflag=direct prevents any kind of buffer cache from getting involved.

None of my cores seem maxed out. The cores dealing with interrupts are mostly idle and all the other cores are waiting on I/O as one would expect.

Cpu0  :  0.0%us,  1.0%sy,  0.0%ni, 91.2%id,  7.5%wa,  0.0%hi,  0.2%si,  0.0%st
Cpu1  :  0.0%us,  0.8%sy,  0.0%ni, 93.0%id,  0.2%wa,  0.0%hi,  6.0%si,  0.0%st
Cpu2  :  0.0%us,  0.6%sy,  0.0%ni, 94.4%id,  0.1%wa,  0.0%hi,  4.8%si,  0.0%st
Cpu3  :  0.0%us,  7.5%sy,  0.0%ni, 36.3%id, 56.1%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  1.3%sy,  0.0%ni, 85.7%id,  4.9%wa,  0.0%hi,  8.1%si,  0.0%st
Cpu5  :  0.1%us,  5.5%sy,  0.0%ni, 36.2%id, 58.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us,  5.0%sy,  0.0%ni, 36.3%id, 58.7%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  5.1%sy,  0.0%ni, 36.3%id, 58.5%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.1%us,  8.3%sy,  0.0%ni, 27.2%id, 64.4%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.1%us,  7.9%sy,  0.0%ni, 36.2%id, 55.8%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.0%us,  7.8%sy,  0.0%ni, 36.2%id, 56.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,  7.3%sy,  0.0%ni, 36.3%id, 56.4%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.0%us,  5.6%sy,  0.0%ni, 33.1%id, 61.2%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  0.1%us,  5.3%sy,  0.0%ni, 36.1%id, 58.5%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  0.0%us,  4.9%sy,  0.0%ni, 36.4%id, 58.7%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.1%us,  5.4%sy,  0.0%ni, 36.5%id, 58.1%wa,  0.0%hi,  0.0%si,  0.0%st

Given all this, the throughput reported by running "dstat 10" is in the range of 2200-2300 MB/sec.

Given the math above I would expect something in the range of 2*1920 ~= 3600+ MB/sec.

Does anybody have any idea where my missing bandwidth went?

Thanks!

I have to add some archive\staging storage to an SLES 10 Server. The requirement is to present fairly large volumes (9-20TB each approx, 60TB or so in total) that will be used to store archive data (literally, this is for a library) comprising large image files (150Meg Tiff's for the most part) and large tarballs. The data will be overwhelmingly biased to read IO, certainly >95% and probably over 99%.

The storage has already been purchased - a Dell MD3000 SAS array daisy chained with 2 MD1000's fully populated with 2TB 7200 RPM SATA drives, 45 drives in total. The stack of arrays is connected using two dual ported external SAS adapters ie there are 4 paths to the stack.

My intention is to configure these as a set of 4 volumes sitting on 4 RAID groups with one hot spare per array. All groups will be RAID 6 with 7 or 14 drives, and each RAID group will be presented as a single LUN using all of the capacity in that group. On the SLES side these need to be formatted as XFS volumes.

I have limited experience with SLES (and Linux in general) and I'm looking for some recommendations about this, specifically:

1. Are there any specific things to be careful about when configuring XFS volumes of this size under SLES 10, ie are default settings going to be OK given the IO profile?
2. What's the best way to initialize\partition\format these? I used Parted to set a Disk label and the YAST Partition Manager (accepting all defaults) to create and format the XFS volume for my initial test.
3. How do I set up multipathing? When I present an initial test volume it appears as four separate devices (/dev/sdl,/dev/sdm,/dev/sdn and /dev/sdn). What do I do to work with this as a single volume?
4. In my initial testing I'm seeing transfer rates from an existing EMC Clariion SAN volume of around 30Meg/sec. This is a lot lower than I'd expect, even accounting for the RAID 6 write penalty I'd expected to see something in the ballpark of 70-100Meg/sec.
5. How can I tell if everything is OK - where should I look for errors\warnings etc? The YAST Partition editor takes a very long time to launch for example and I'd like to understand why.
6. Would you partition this differently and\or use a different filesystem and if so why?

The server is a Dell 2950 - I haven't checked the detailed specs but top shows utilization hovering in the low single digits at most.