Questions[mpio](server)

I am building a new sub-cluster, thereby I am facing few questions.

1. Which technology is more efficient for "boosting throughput" ? (Will be great if you can provide any comparison) As far as I done my google job, there are two technology's which provides additional paths inside one NIC. Link Aggregation Control Protocol and Multipath I/O. It would be great if some one can provide me with any benefits or drawbacks of these technology.

(I am asking the second qestion from curiosity , just want to hear your opinion)

2. Which connection is preferable (Ethernet of FC)?

I heared that Ethernet connection is slower when it comes to comparing Ethernet and FC.

Recent owner of a Dell PowerVault MD3600i i'm experiencing some weird results.

I have a dedicated 24x 10GbE Switch (PowerConnect 8024), setup to jumbo frames 9K.

The MD3600 has 2 RAID controllers, each has 2x 10GbE ethernet nics. There's nothing else on the switch; one VLAN for SAN traffic.

Here's my multipath.conf

defaults {
    udev_dir        /dev
    polling_interval    5
    selector        "round-robin 0"
    path_grouping_policy    multibus
    getuid_callout      "/sbin/scsi_id -g -u -s /block/%n"
    prio_callout        none
    path_checker        readsector0
    rr_min_io       100
    max_fds         8192
    rr_weight       priorities
    failback        immediate
    no_path_retry       fail
    user_friendly_names yes
#   prio            rdac
}
blacklist {
    device {
               vendor "*"
        product "Universal Xport"
        }
#   devnode "^sd[a-z]"
}

devices {
    device {
           vendor "DELL"
           product "MD36xxi"
           path_grouping_policy group_by_prio
           prio rdac 
        #  polling_interval  5
           path_checker rdac
           path_selector "round-robin 0"
           hardware_handler "1 rdac"
           failback immediate
           features "2 pg_init_retries 50"
           no_path_retry 30
           rr_min_io 100
           prio_callout "/sbin/mpath_prio_rdac /dev/%n"
       }
}

And iscsid.conf :

node.startup = automatic
node.session.timeo.replacement_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 10
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144

After my tests; i can barely come to 200 Mb/s read/write.

Should I expect more than that ? Providing it has dual 10 GbE my thoughts where to come around the 400 Mb/s.

Any ideas ? Guidelines ? Troubleshooting tips ?

EDIT:

The array is setup as a single logical volume of 5.7TB Disks are all 1TB 7.2k SAS 6GB (ST1000NM0001) RAID is RAID10

Some lines of the Swith Configuration:

interface Te1/0/23
storm-control broadcast
storm-control multicast
spanning-tree portfast
mtu 9000
switchport access vlan 40
exit
...
iscsi cos vpt 5
management access-list "default"
permit service ssh priority 1
permit service http priority 2
permit service https priority 3

And multipath output:

[root@xnode4 ~]# multipath -ll -v2
multipath.conf line 30, invalid keyword: prio
mpath1 (36d4ae520009bd7cc0000030e4fe8230b) dm-2 DELL,MD36xxi
[size=5.5T][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=400][active]
 \_ 7:0:0:0   sdc 8:32  [active][ready]
 \_ 9:0:0:0   sde 8:64  [active][ready]
 \_ 11:0:0:0  sdi 8:128 [active][ready]
 \_ 13:0:0:0  sdn 8:208 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 10:0:0:0  sdj 8:144 [active][ghost]
 \_ 12:0:0:0  sdh 8:112 [active][ghost]
 \_ 8:0:0:0   sdd 8:48  [active][ghost]
 \_ 6:0:0:0   sdb 8:16  [active][ghost]

I have a BL460c G7 blade server with OS Windows 2008 R2 SP1. This is a brand new C7000 enclosure, with FlexFabric interconnects. I got my FC switches setup and zoned properly to our Clariion CX4, and can see all the hosts that are assigned FCoE HBAs on both paths in both Navisphere and in HP Virtual Connect Manager.

So I went ahead and created a storage group for a test server, assigned the appropriate host, assigned the LUN to the server. So far so good, log onto server and I can see 4 unknown disks.... No problem, I install MS MPIO, no luck, can't initialize the disks, and the multiple disks don't go away. Still no problem, I install PowerPath version 5.5 reboot. Now I see 3 disks. One is initialized and ready to go, but I still have 2 disks that I can't initialize, can't offline, can't delete. If I right click in storage manager and go to properties I can see that the MS MPIO tab, but I can't make a path active.

I want to get rid of these phantom disks, but so far nothing is working and google searches are showing up some odd results, so obviously I'm not framing my question right. I thought I'd ask here real quick. Does anyone know a quick way to get rid of these unknown disks.

Another question, do I need the MPIO feature installed if I have PowerPath installed? This is my first time installing Windows 2008 R2 in this fashion and I'm not sure if that feature is needed or not right now.

So some more information to add to this. It seems I'm dealing with more of a Windows issue than anything else. I removed the LUN from the server, uninstalled PowerPath completely, removed the MPIO feature from the server, and rebooted twice. Now I am back to the original 4 Unknown Disks (plus the local Disk 0 containing the OS partition of course, which is working fine)

I went to diskpart, I could see all 4 Unknown disks, I selected each disk, ran clean (just in case i'd somehow brought them online previously as GPT and didn't realize it) After a few minutes I was no longer able to see the disks when I ran list disk. However, the disks are still in Disk Management. When I try and offline the disks from Disk Management I get an error: Virtual Disk Manager - The system cannot find the file specified. Accompanied by an error in System Event Logs:

Log Name:      System
Source:        Virtual Disk Service
Date:          6/25/2012 4:02:01 PM
Event ID:      1
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      hostname.local
Description:
Unexpected failure. Error code: 2@02000018
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Virtual Disk Service" />
    <EventID Qualifiers="49664">1</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-06-25T21:02:01.000000000Z" />
    <EventRecordID>4239</EventRecordID>
    <Channel>System</Channel>
    <Computer>hostname.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>2@02000018</Data>
  </EventData>
</Event>

I feel sure there is a place I can go in the Registry to get rid of these, I just can't recall where and I am loathe to experiement.

So to recap, there are currently no LUNS attached at all, I still have the phantom disks, and I'm getting The system cannot find the file specified from Virtual Disk Manager when I try to take them offline.

Thanks!

I have a desire to pass iSCSI traffic between my Windows 2K8 Server lab system and my NetApp filer across two separate network stacks*.

My configuration is as follows:

• one Win2K8 server with the iSCSI software initiator installed, the MPIO component installed, and two network interfaces: 192.168.201.85/24 and 192.168.202.85/24
• one NetApp filer with a LUN published to the Windows server's IQN, and two interfaces: 192.168.201.200/24 and 192.168.202.200/24
• two separate switches, one for 192.168.201.0/24 and one for 192.168.202.0/24. Both are flat (un-VLAN'd) and are not connected to any other network equipment -- including each other.

I have configured the MPIO component to register the iSCSI software initiator "adapter" class.

Then I have gone into the the iSCSI initiator control panel and added both filer addresses as "targets" and run discovery against them. This shows the single LUN available.

I have then "logged on" to the LUN twice, selecting a different "source" IP address for each connection. Both connections have "re-connect at boot" checked and "MPIO" checked.

When I examine the target, I see two connections to the target, one for each IP address that the NetApp is using.

When I examine my persistent connections, I see two connections, one for each IP address that the NetApp is using.

(I should mention at this point that I have tested both filer IPs by demonstrating a single connection to each IP, mounting and then using a drive across that IP.)

I then go into my Disk Mangler and set up the partition on the LUN, and mark it Online. The disk works as expected.

Now I go into the new disk's properties, and click the MPIO tab. I can see two connections in use for this disk. However I don't know how to associate the connection I see in this tab with the connection I see in the iSCSI initiator screens -- so while I presume there's one connection for each connection in the iSCSI initiator screen, I can't prove it.

In the MPIO tab, I have several options.

I have reduced the timers all down to 1 second each and enabled the Path Verification. So my understanding of these settings mean:

• each second the Windows server will verify that the path is valid ie the remote target IP is answering properly;
• the server will retry only once after a failure is detected, one second after the failure is detected;
• the server will mark as invalid and remove the path one second after a failure.

Regarding redundancy, there are a couple things I have tried:

• If I set up both connections as Active/Active and select Round Robin use, the disk works. If I set up a copy operation on the disk and simulate a network failure by pulling one of the network cables out, the connection stops for ~30 seconds and then keeps going.
• If I set up the connection as a Failover-only by marking one connection as Standby/Passive and selecting Failover-only, again the connection works. (Interestingly, disk-to-disk copies appear to consistently flow at about twice the speed of the Round Robin, but anyways.) If I simulate a failure by pulling the standby cable out, the connection stops for about 1 second and then keeps going. If I simulate a failure by pulling the Active cable out, the connection stops -- and I can't ping the filer across either wire. Eventually the OS tells me the disk has failed. The network stays in this state for several hours (after which time I got tired of waiting on it and rebooted the server).

I did some research and found a Microsoft KB 968287 which talks about failover not completing because of a counter error in the MPIO.sys driver in Win2K8 and Vista, but installing this hotfix has not changed anything that I can see.

All this makes me suspect I'm missing something fundamental. Am I doing this wrong?

The real goal here is to provide a more-reliable iSCSI transport over which to run VMs and mount Exchange stores on my Hyper-V cluster. We do know that Exchange in particular will unmount information stores very quickly if a disk hiccup is detected, so we were hoping that MPIO would permit data to flow even if one path failed.

*= We currently have a single iSCSI switch, but when that started to misbehave we had to take down our entire world in order to flash the firmware on the one switch. Therefore we want two totally isolated network paths -- NICs, switches, and interfaces on the other end -- so that we can take half of them out of service at any given time for maintenance without killing the world.

My new database server (Windows Server 2008 R2) has a 4 Gbps connection to my Dell EqualLogic iSCSI SAN. I've been testing it by backing up multiple databases at the same time. There appears to be a bottleneck with the network adapter. It shows up as 4 1 Gbps connections, and the network utilization never exceeds 2 Gbps. Any recommendations where to look next?

SAN: 48 drives. 10K 600GB SAS. Raid 50.

Update:

Flow control is set to auto, but I think it is being used because MPIO is installed. There is a switch and I'm working on jumbo frames, but while researching MPIO I noticed something that might be affecting performance.

In the database backup test I was performing, 4 volumes were involved: one for data, one for the log, and 2 for the backup files (split by RedGate SQL Backup to improve performance). I was trying to determine if there was any performance benefit to separate volumes on a SAN. On my old server, I would use 4 different RAID arrays in this situation.

Each of my 1 Gbps network adapters has an IP assigned in the range 10.0.1.251-10.0.1.254. Each volume has 2 of these IP addresses assigned to it in the "Dell EqualLogic MPIO" tab in iSCSI Initiator Properties. 251 & 252 are used for the log & both backup volumes. 252 & 253 are used for the data volume. It seems like this could be a limiting factor.

I'm going to see if I can assign them different IP addresses.

Update 2:

Using the MPIO Remote Setup Wizard, I changed the "Max sessions per volume slice" from the default 2 to 4 and rebooted. Each volume now has 4 paths, one for each NIC. The utilization is now fairly evenly distributed between the 4 NICs, but the total is still less than 2 Gbps. Each one maxes out at about 45%. Once I get access to my switch I will try to get jumbo frames working and see if that helps.

Solution:

Jumbo Frames. They had to be set on the database server NICs and on the switch. The SAN handled it automatically. This required a reboot on the DB server.