Questions[dell-powervault](server)

Recent owner of a Dell PowerVault MD3600i i'm experiencing some weird results.

I have a dedicated 24x 10GbE Switch (PowerConnect 8024), setup to jumbo frames 9K.

The MD3600 has 2 RAID controllers, each has 2x 10GbE ethernet nics. There's nothing else on the switch; one VLAN for SAN traffic.

Here's my multipath.conf

defaults {
    udev_dir        /dev
    polling_interval    5
    selector        "round-robin 0"
    path_grouping_policy    multibus
    getuid_callout      "/sbin/scsi_id -g -u -s /block/%n"
    prio_callout        none
    path_checker        readsector0
    rr_min_io       100
    max_fds         8192
    rr_weight       priorities
    failback        immediate
    no_path_retry       fail
    user_friendly_names yes
#   prio            rdac
}
blacklist {
    device {
               vendor "*"
        product "Universal Xport"
        }
#   devnode "^sd[a-z]"
}

devices {
    device {
           vendor "DELL"
           product "MD36xxi"
           path_grouping_policy group_by_prio
           prio rdac 
        #  polling_interval  5
           path_checker rdac
           path_selector "round-robin 0"
           hardware_handler "1 rdac"
           failback immediate
           features "2 pg_init_retries 50"
           no_path_retry 30
           rr_min_io 100
           prio_callout "/sbin/mpath_prio_rdac /dev/%n"
       }
}

And iscsid.conf :

node.startup = automatic
node.session.timeo.replacement_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 10
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144

After my tests; i can barely come to 200 Mb/s read/write.

Should I expect more than that ? Providing it has dual 10 GbE my thoughts where to come around the 400 Mb/s.

Any ideas ? Guidelines ? Troubleshooting tips ?

EDIT:

The array is setup as a single logical volume of 5.7TB Disks are all 1TB 7.2k SAS 6GB (ST1000NM0001) RAID is RAID10

Some lines of the Swith Configuration:

interface Te1/0/23
storm-control broadcast
storm-control multicast
spanning-tree portfast
mtu 9000
switchport access vlan 40
exit
...
iscsi cos vpt 5
management access-list "default"
permit service ssh priority 1
permit service http priority 2
permit service https priority 3

And multipath output:

[root@xnode4 ~]# multipath -ll -v2
multipath.conf line 30, invalid keyword: prio
mpath1 (36d4ae520009bd7cc0000030e4fe8230b) dm-2 DELL,MD36xxi
[size=5.5T][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=400][active]
 \_ 7:0:0:0   sdc 8:32  [active][ready]
 \_ 9:0:0:0   sde 8:64  [active][ready]
 \_ 11:0:0:0  sdi 8:128 [active][ready]
 \_ 13:0:0:0  sdn 8:208 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 10:0:0:0  sdj 8:144 [active][ghost]
 \_ 12:0:0:0  sdh 8:112 [active][ghost]
 \_ 8:0:0:0   sdd 8:48  [active][ghost]
 \_ 6:0:0:0   sdb 8:16  [active][ghost]

I have an MD3000i Dell SAN with dual controllers. Long story short - it managed to muck up some of its own readings on drives and Dell's support told me I could try to shake it loose with a firmware and NVSRAM upgrade which I need anyway. Since I have a dual controller this can apparently be done live and the upgrade process handles shuffling LUNs between controllers as it updates.

Does anyone have experience with this? The Dell technician claimed it could be done and the documentation suggests as much for dual-controller configurations, but it makes me very nervous to do it.

My main questions are:

1. Has anyone done this successfully or know someone who has?
2. Are there any pitfalls to be aware of?

Alright. First and foremost, Warning. This is a bigger-then-normal question. I like to be thorough and try to eliminate all possible "easymode" answers, as well as give everyone a feel of what i've tried. I've included several images of our setup and the problem it is having..

TLDR Version: So I've followed the guides located here: ESX Deployment Guide V1 this is the guide Dell has sent me to setup two ESX3.5 servers mounting a Dell MD3000i. It doesn't work. Both servers can't use the same storage partition on the MD3000. Both servers see it, but only one server can actually use it. (that server being whatever server created the partition on the target.) Both ESX servers are members of the Host Group.

Full Version

I have 2 ESX3.5 Servers (10.0.7.102, also called EPI2, and 10.0.7.103, also called EPI3.) connected to a iSCSI SAN Device (Dell MD3000i). Both ESX servers can "scan" the SAN and see the LUNS.

Part One: MD3000i Storage

On the MD3000i, Both servers are in my host group.

I have two partitions, VM1 and VM2, both 1.6TB (vmware doesn't like anything past 2tb.)

And you can even see that the ESX servers are targetting the MD3000 just fine.

Part Two: The ESX Servers

Figure 1.

So as you can see above, Both ESX Servers (10.0.7.102 and 10.0.7.103) are able to see and scan the MD3000i SAN.
Figure 2.

Above is the storage both servers see. I created the storage partition on EPI2 (102). I then Extended the partition to include the second LUN for a grand total of 3.27 TB of storage.

When i "rescan" on 103 (the server not mounting the partition), I get the below log in log/messages. Mar 11 10:41:18 epi3 kernel: scsi1: remove-single-device 0 0 0 failed, device busy(4). being the only line that grabs my attentions. (EPI3 is the server name)

Mar 11 10:41:04 epi3 vmkiscsid[5436]: Connected to Discovery Address 192.168.130.101 
Mar 11 10:41:04 epi3 vmkiscsid[5437]: Connected to Discovery Address 192.168.130.102 
Mar 11 10:41:04 epi3 vmkiscsid[5438]: Connected to Discovery Address 192.168.131.101 
Mar 11 10:41:04 epi3 vmkiscsid[5439]: Connected to Discovery Address 192.168.131.102 
Mar 11 10:41:17 epi3 kernel: scsi singledevice 2 0 0 0
Mar 11 10:41:17 epi3 kernel:   Vendor: DELL      Model: MD3000i           Rev: 0735
Mar 11 10:41:17 epi3 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar 11 10:41:17 epi3 kernel: VMWARE SCSI Id: Supported VPD pages for sdb : 0x0 0x80 0x83 0x85 0x86 0x87 0xc0 0xc1 0xc2 0xc3 0xc4 0xc8 0xc9 0xca 0xd0 
Mar 11 10:41:17 epi3 kernel: VMWARE SCSI Id: Device id info for sdb: 0x1 0x3 0x0 0x10 0x60 0x1 0xe4 0xf0 0x0 0x1a 0x1a 0xa2 0x0 0x0 0x15 0xe2 0x4d 0x75 0xf6 0x99 0x53 0x98 0x0 0x54 0x69 0x71 0x6e 0x2e 0x31 0x39 0x38 0x34 0x2d 0x30 0x35 0x2e 0x63 0x6f 0x6d 0x2e 0x64 0x65 0x6c 0x6c 0x3a 0x70 0x6f 0x77 0x65 0x72 0x76 0x61 0x75 0x6c 0x74 0x2e 0x36 0x30 0x30 0x31 0x65 0x34 0x66 0x30 0x30 0x30 0x31 0x61 0x31 0x61 0x61 0x32 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x34 0x37 0x39 0x30 0x36 0x32 0x32 0x65 0x2c 0x74 0x2c 0x30 0x78 0x30 0x30 0x30 0x31 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x32 0x0 0x0 0x0 0x51 0x94 0x0 0x4 0x0 0x0 0x80 0x1 0x53 0xa8 0x0 0x44 0x69 0x71 0x6e 0x2e 0x31 0x39 0x38 0x34 0x2d 0x30 0x35 0x2e 0x63 0x6f 0x6d 0x2e 0x64 0x65 0x6c 0x6c 0x3a 0x70 0x6f 0x77 0x65 0x72 0x76 0x61 0x75 0x6c 0x74 0x2e 0x36 0x30 0x30 0x31 0x65 0x34 0x66 0x30 0x30 0x30 0x31 0x61 0x31 0x61 0x61 0x32 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x34 0x37 0x39 0x30 0x36 0x32 0x32 0x65 0x0 0x0 0x0 0x0 
Mar 11 10:41:17 epi3 kernel: VMWARE SCSI Id: Id for sdb 0x60 0x01 0xe4 0xf0 0x00 0x1a 0x1a 0xa2 0x00 0x00 0x15 0xe2 0x4d 0x75 0xf6 0x99 0x4d 0x44 0x33 0x30 0x30 0x30 
Mar 11 10:41:17 epi3 kernel: VMWARE: Unique Device attached as scsi disk sdb at scsi2, channel 0, id 0, lun 0
Mar 11 10:41:17 epi3 kernel: Attached scsi disk sdb at scsi2, channel 0, id 0, lun 0
Mar 11 10:41:17 epi3 kernel: scan_scsis starting finish
Mar 11 10:41:17 epi3 kernel: SCSI device sdb: 3509329920 512-byte hdwr sectors (1797751 MB)
Mar 11 10:41:17 epi3 kernel:  sdb: sdb1
Mar 11 10:41:17 epi3 kernel: scan_scsis done with finish
Mar 11 10:41:17 epi3 kernel: scsi singledevice 2 0 0 1
Mar 11 10:41:17 epi3 kernel:   Vendor: DELL      Model: MD3000i           Rev: 0735
Mar 11 10:41:17 epi3 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar 11 10:41:18 epi3 kernel: VMWARE SCSI Id: Supported VPD pages for sdc : 0x0 0x80 0x83 0x85 0x86 0x87 0xc0 0xc1 0xc2 0xc3 0xc4 0xc8 0xc9 0xca 0xd0 
Mar 11 10:41:18 epi3 kernel: VMWARE SCSI Id: Device id info for sdc: 0x1 0x3 0x0 0x10 0x60 0x1 0xe4 0xf0 0x0 0x1a 0x1a 0x86 0x0 0x0 0xd 0xb7 0x4d 0x75 0xf2 0x77 0x53 0x98 0x0 0x54 0x69 0x71 0x6e 0x2e 0x31 0x39 0x38 0x34 0x2d 0x30 0x35 0x2e 0x63 0x6f 0x6d 0x2e 0x64 0x65 0x6c 0x6c 0x3a 0x70 0x6f 0x77 0x65 0x72 0x76 0x61 0x75 0x6c 0x74 0x2e 0x36 0x30 0x30 0x31 0x65 0x34 0x66 0x30 0x30 0x30 0x31 0x61 0x31 0x61 0x61 0x32 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x34 0x37 0x39 0x30 0x36 0x32 0x32 0x65 0x2c 0x74 0x2c 0x30 0x78 0x30 0x30 0x30 0x31 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x32 0x0 0x0 0x0 0x51 0x94 0x0 0x4 0x0 0x0 0x80 0x1 0x53 0xa8 0x0 0x44 0x69 0x71 0x6e 0x2e 0x31 0x39 0x38 0x34 0x2d 0x30 0x35 0x2e 0x63 0x6f 0x6d 0x2e 0x64 0x65 0x6c 0x6c 0x3a 0x70 0x6f 0x77 0x65 0x72 0x76 0x61 0x75 0x6c 0x74 0x2e 0x36 0x30 0x30 0x31 0x65 0x34 0x66 0x30 0x30 0x30 0x31 0x61 0x31 0x61 0x61 0x32 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x34 0x37 0x39 0x30 0x36 0x32 0x32 0x65 0x0 0x0 0x0 0x0 
Mar 11 10:41:18 epi3 kernel: VMWARE SCSI Id: Id for sdc 0x60 0x01 0xe4 0xf0 0x00 0x1a 0x1a 0x86 0x00 0x00 0x0d 0xb7 0x4d 0x75 0xf2 0x77 0x4d 0x44 0x33 0x30 0x30 0x30 
Mar 11 10:41:18 epi3 kernel: VMWARE: Unique Device attached as scsi disk sdc at scsi2, channel 0, id 0, lun 1
Mar 11 10:41:18 epi3 kernel: Attached scsi disk sdc at scsi2, channel 0, id 0, lun 1
Mar 11 10:41:18 epi3 kernel: scan_scsis starting finish
Mar 11 10:41:18 epi3 kernel: SCSI device sdc: 3509329920 512-byte hdwr sectors (1797751 MB)
Mar 11 10:41:18 epi3 kernel:  sdc: sdc1
Mar 11 10:41:18 epi3 kernel: scan_scsis done with finish
Mar 11 10:41:18 epi3 kernel: scsi1: remove-single-device 0 0 0 failed, device busy(4).
Mar 11 10:41:18 epi3 kernel: scsi singledevice 1 0 0 0

Things I've Tried:

1. Removing iSCSI targets from only 103, disabling iSCSI, rebooting, enabled iSCSI, re-adding targets, rescan. Same result.
2. Removing partition on 102, Formatted partition on 103 instead. Same result, except flipped. 103 can use storage, 102 can not.
3. Starting Over. Removing all iSCSI Targets on both ESX Boxes, disabling iSCSI, turning off the firewall for iSCSI, rebooting ESX. Then on the MD3000, Removed the Host Group, Removed the Host-to-Virtual Mappings, Restarted the SAN. Followed the Documentation again, same result. Both servers see the storage, but only one server can use it.
4. Disabling and Re-enabling VMware DRS and HA. Same result.
5. Flat-out turning off VMware DRS and HA, and doing the "start over" step to see if maybe that borked it. Same Result.

I'm kinda loosing my mind here, Everything i read online says "just partition it and if the ESX boxes can see the targets, it just works".... well crap.

Any ideas, any other things to try? Can anyone atleast point me in the right direction? I'm really tired of working from 1am til 4am (our maintenance hours)

I've a MD3000 and the max partition size that I can allocate is 2TB. Is there any way, through firmware updates or other trickery, to allocate a useful amount of space to a partition, 4TB say?

Is it possible to use a different PERC card with the MD3000 to do this?

I have to add some archive\staging storage to an SLES 10 Server. The requirement is to present fairly large volumes (9-20TB each approx, 60TB or so in total) that will be used to store archive data (literally, this is for a library) comprising large image files (150Meg Tiff's for the most part) and large tarballs. The data will be overwhelmingly biased to read IO, certainly >95% and probably over 99%.

The storage has already been purchased - a Dell MD3000 SAS array daisy chained with 2 MD1000's fully populated with 2TB 7200 RPM SATA drives, 45 drives in total. The stack of arrays is connected using two dual ported external SAS adapters ie there are 4 paths to the stack.

My intention is to configure these as a set of 4 volumes sitting on 4 RAID groups with one hot spare per array. All groups will be RAID 6 with 7 or 14 drives, and each RAID group will be presented as a single LUN using all of the capacity in that group. On the SLES side these need to be formatted as XFS volumes.

I have limited experience with SLES (and Linux in general) and I'm looking for some recommendations about this, specifically:

1. Are there any specific things to be careful about when configuring XFS volumes of this size under SLES 10, ie are default settings going to be OK given the IO profile?
2. What's the best way to initialize\partition\format these? I used Parted to set a Disk label and the YAST Partition Manager (accepting all defaults) to create and format the XFS volume for my initial test.
3. How do I set up multipathing? When I present an initial test volume it appears as four separate devices (/dev/sdl,/dev/sdm,/dev/sdn and /dev/sdn). What do I do to work with this as a single volume?
4. In my initial testing I'm seeing transfer rates from an existing EMC Clariion SAN volume of around 30Meg/sec. This is a lot lower than I'd expect, even accounting for the RAID 6 write penalty I'd expected to see something in the ballpark of 70-100Meg/sec.
5. How can I tell if everything is OK - where should I look for errors\warnings etc? The YAST Partition editor takes a very long time to launch for example and I'd like to understand why.
6. Would you partition this differently and\or use a different filesystem and if so why?

The server is a Dell 2950 - I haven't checked the detailed specs but top shows utilization hovering in the low single digits at most.