Home / server / Questions / 246320
Accepted
grufftech
Asked: 2011-03-12 10:03:47 +0800 CST

ESX3.5 Cluster & MD3000i -- Both servers see iSCSI Targets, Only one server can use partition

  • 772

Alright. First and foremost, Warning. This is a bigger-then-normal question. I like to be thorough and try to eliminate all possible "easymode" answers, as well as give everyone a feel of what i've tried. I've included several images of our setup and the problem it is having..

TLDR Version: So I've followed the guides located here: ESX Deployment Guide V1 this is the guide Dell has sent me to setup two ESX3.5 servers mounting a Dell MD3000i. It doesn't work. Both servers can't use the same storage partition on the MD3000. Both servers see it, but only one server can actually use it. (that server being whatever server created the partition on the target.) Both ESX servers are members of the Host Group.

Full Version

I have 2 ESX3.5 Servers (10.0.7.102, also called EPI2, and 10.0.7.103, also called EPI3.) connected to a iSCSI SAN Device (Dell MD3000i). Both ESX servers can "scan" the SAN and see the LUNS.

Part One: MD3000i Storage

On the MD3000i, Both servers are in my host group. enter image description here

I have two partitions, VM1 and VM2, both 1.6TB (vmware doesn't like anything past 2tb.) enter image description here

And you can even see that the ESX servers are targetting the MD3000 just fine.enter image description here

Part Two: The ESX Servers

scanning the md3000 Figure 1.

So as you can see above, Both ESX Servers (10.0.7.102 and 10.0.7.103) are able to see and scan the MD3000i SAN.
Storage on Vmware Figure 2.

Above is the storage both servers see. I created the storage partition on EPI2 (102). I then Extended the partition to include the second LUN for a grand total of 3.27 TB of storage.

vmwarepartition

When i "rescan" on 103 (the server not mounting the partition), I get the below log in log/messages. Mar 11 10:41:18 epi3 kernel: scsi1: remove-single-device 0 0 0 failed, device busy(4). being the only line that grabs my attentions. (EPI3 is the server name)

Mar 11 10:41:04 epi3 vmkiscsid[5436]: Connected to Discovery Address 192.168.130.101 
Mar 11 10:41:04 epi3 vmkiscsid[5437]: Connected to Discovery Address 192.168.130.102 
Mar 11 10:41:04 epi3 vmkiscsid[5438]: Connected to Discovery Address 192.168.131.101 
Mar 11 10:41:04 epi3 vmkiscsid[5439]: Connected to Discovery Address 192.168.131.102 
Mar 11 10:41:17 epi3 kernel: scsi singledevice 2 0 0 0
Mar 11 10:41:17 epi3 kernel:   Vendor: DELL      Model: MD3000i           Rev: 0735
Mar 11 10:41:17 epi3 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar 11 10:41:17 epi3 kernel: VMWARE SCSI Id: Supported VPD pages for sdb : 0x0 0x80 0x83 0x85 0x86 0x87 0xc0 0xc1 0xc2 0xc3 0xc4 0xc8 0xc9 0xca 0xd0 
Mar 11 10:41:17 epi3 kernel: VMWARE SCSI Id: Device id info for sdb: 0x1 0x3 0x0 0x10 0x60 0x1 0xe4 0xf0 0x0 0x1a 0x1a 0xa2 0x0 0x0 0x15 0xe2 0x4d 0x75 0xf6 0x99 0x53 0x98 0x0 0x54 0x69 0x71 0x6e 0x2e 0x31 0x39 0x38 0x34 0x2d 0x30 0x35 0x2e 0x63 0x6f 0x6d 0x2e 0x64 0x65 0x6c 0x6c 0x3a 0x70 0x6f 0x77 0x65 0x72 0x76 0x61 0x75 0x6c 0x74 0x2e 0x36 0x30 0x30 0x31 0x65 0x34 0x66 0x30 0x30 0x30 0x31 0x61 0x31 0x61 0x61 0x32 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x34 0x37 0x39 0x30 0x36 0x32 0x32 0x65 0x2c 0x74 0x2c 0x30 0x78 0x30 0x30 0x30 0x31 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x32 0x0 0x0 0x0 0x51 0x94 0x0 0x4 0x0 0x0 0x80 0x1 0x53 0xa8 0x0 0x44 0x69 0x71 0x6e 0x2e 0x31 0x39 0x38 0x34 0x2d 0x30 0x35 0x2e 0x63 0x6f 0x6d 0x2e 0x64 0x65 0x6c 0x6c 0x3a 0x70 0x6f 0x77 0x65 0x72 0x76 0x61 0x75 0x6c 0x74 0x2e 0x36 0x30 0x30 0x31 0x65 0x34 0x66 0x30 0x30 0x30 0x31 0x61 0x31 0x61 0x61 0x32 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x34 0x37 0x39 0x30 0x36 0x32 0x32 0x65 0x0 0x0 0x0 0x0 
Mar 11 10:41:17 epi3 kernel: VMWARE SCSI Id: Id for sdb 0x60 0x01 0xe4 0xf0 0x00 0x1a 0x1a 0xa2 0x00 0x00 0x15 0xe2 0x4d 0x75 0xf6 0x99 0x4d 0x44 0x33 0x30 0x30 0x30 
Mar 11 10:41:17 epi3 kernel: VMWARE: Unique Device attached as scsi disk sdb at scsi2, channel 0, id 0, lun 0
Mar 11 10:41:17 epi3 kernel: Attached scsi disk sdb at scsi2, channel 0, id 0, lun 0
Mar 11 10:41:17 epi3 kernel: scan_scsis starting finish
Mar 11 10:41:17 epi3 kernel: SCSI device sdb: 3509329920 512-byte hdwr sectors (1797751 MB)
Mar 11 10:41:17 epi3 kernel:  sdb: sdb1
Mar 11 10:41:17 epi3 kernel: scan_scsis done with finish
Mar 11 10:41:17 epi3 kernel: scsi singledevice 2 0 0 1
Mar 11 10:41:17 epi3 kernel:   Vendor: DELL      Model: MD3000i           Rev: 0735
Mar 11 10:41:17 epi3 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Mar 11 10:41:18 epi3 kernel: VMWARE SCSI Id: Supported VPD pages for sdc : 0x0 0x80 0x83 0x85 0x86 0x87 0xc0 0xc1 0xc2 0xc3 0xc4 0xc8 0xc9 0xca 0xd0 
Mar 11 10:41:18 epi3 kernel: VMWARE SCSI Id: Device id info for sdc: 0x1 0x3 0x0 0x10 0x60 0x1 0xe4 0xf0 0x0 0x1a 0x1a 0x86 0x0 0x0 0xd 0xb7 0x4d 0x75 0xf2 0x77 0x53 0x98 0x0 0x54 0x69 0x71 0x6e 0x2e 0x31 0x39 0x38 0x34 0x2d 0x30 0x35 0x2e 0x63 0x6f 0x6d 0x2e 0x64 0x65 0x6c 0x6c 0x3a 0x70 0x6f 0x77 0x65 0x72 0x76 0x61 0x75 0x6c 0x74 0x2e 0x36 0x30 0x30 0x31 0x65 0x34 0x66 0x30 0x30 0x30 0x31 0x61 0x31 0x61 0x61 0x32 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x34 0x37 0x39 0x30 0x36 0x32 0x32 0x65 0x2c 0x74 0x2c 0x30 0x78 0x30 0x30 0x30 0x31 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x32 0x0 0x0 0x0 0x51 0x94 0x0 0x4 0x0 0x0 0x80 0x1 0x53 0xa8 0x0 0x44 0x69 0x71 0x6e 0x2e 0x31 0x39 0x38 0x34 0x2d 0x30 0x35 0x2e 0x63 0x6f 0x6d 0x2e 0x64 0x65 0x6c 0x6c 0x3a 0x70 0x6f 0x77 0x65 0x72 0x76 0x61 0x75 0x6c 0x74 0x2e 0x36 0x30 0x30 0x31 0x65 0x34 0x66 0x30 0x30 0x30 0x31 0x61 0x31 0x61 0x61 0x32 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x30 0x34 0x37 0x39 0x30 0x36 0x32 0x32 0x65 0x0 0x0 0x0 0x0 
Mar 11 10:41:18 epi3 kernel: VMWARE SCSI Id: Id for sdc 0x60 0x01 0xe4 0xf0 0x00 0x1a 0x1a 0x86 0x00 0x00 0x0d 0xb7 0x4d 0x75 0xf2 0x77 0x4d 0x44 0x33 0x30 0x30 0x30 
Mar 11 10:41:18 epi3 kernel: VMWARE: Unique Device attached as scsi disk sdc at scsi2, channel 0, id 0, lun 1
Mar 11 10:41:18 epi3 kernel: Attached scsi disk sdc at scsi2, channel 0, id 0, lun 1
Mar 11 10:41:18 epi3 kernel: scan_scsis starting finish
Mar 11 10:41:18 epi3 kernel: SCSI device sdc: 3509329920 512-byte hdwr sectors (1797751 MB)
Mar 11 10:41:18 epi3 kernel:  sdc: sdc1
Mar 11 10:41:18 epi3 kernel: scan_scsis done with finish
Mar 11 10:41:18 epi3 kernel: scsi1: remove-single-device 0 0 0 failed, device busy(4).
Mar 11 10:41:18 epi3 kernel: scsi singledevice 1 0 0 0

Things I've Tried:

  1. Removing iSCSI targets from only 103, disabling iSCSI, rebooting, enabled iSCSI, re-adding targets, rescan. Same result.
  2. Removing partition on 102, Formatted partition on 103 instead. Same result, except flipped. 103 can use storage, 102 can not.
  3. Starting Over. Removing all iSCSI Targets on both ESX Boxes, disabling iSCSI, turning off the firewall for iSCSI, rebooting ESX. Then on the MD3000, Removed the Host Group, Removed the Host-to-Virtual Mappings, Restarted the SAN. Followed the Documentation again, same result. Both servers see the storage, but only one server can use it.
  4. Disabling and Re-enabling VMware DRS and HA. Same result.
  5. Flat-out turning off VMware DRS and HA, and doing the "start over" step to see if maybe that borked it. Same Result.

I'm kinda loosing my mind here, Everything i read online says "just partition it and if the ESX boxes can see the targets, it just works".... well crap.

Any ideas, any other things to try? Can anyone atleast point me in the right direction? I'm really tired of working from 1am til 4am (our maintenance hours)

dell iscsi vmware-esx dell-powervault

4 Answers

  1. I feel your pain....I've done battle with ESX and iSCSI multiple times over the past year.

    I'm not certain, but you might be running into an issue due to the size of the resultant datastore. There's a 2TB limit to an iSCSI LUN, which is fine since you have split it into two 1.6 TB LUNs.

    I wonder if epi3 can't load the datastore because it believes it to be an invalid size.

    Have you tried loading each lun as it's own datastore to see if the hosts can see them correctly that way?

    • 1

  2. Seems like it's allowed iSCSI access but no read/write... Has this been done?

    Select “Yes: This host will share access to the same virtual disks with other hosts”

    (from http://www.dell.com/downloads/global/solutions/pvault_esx_storage_deployment_guide_v1.pdf)

    EDIT: To eliminate ESX as being the issue, can you put the second ESX in a separate host group, and assign that hostgroup a lun? Also, I saw some old posts where if the initiator name was longer than 31 characters, the ESX box would not connect. From what I see on your screenshots, and assuming they fixed that, you should be okay. Just thought it was worth mentioning here.

    • 1
  3. Best Answer

    Its not a great answer, but we solved the problem.

    It seems our "EPI2" server was freaking out in some manner that refused to share its storage.

    Once i removed EPI2 from the cluster and rescanned using EPI1 (ESX4.1) and EPI3 (ESX3.5) both found and mounted the storage properly.

    Since EPI2 had caused these problems we decided to migrate all virtuals off it and upgrade it to 4.1.

    Since upgrading, we've had no problems, all 3 ESX boxes see the storage and share it properly.

    Thanks everyone for your help.

    • 1

  4. The size of an iSCSI disk is limited to 2 TB. http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_config_max.pdf It looks like you are running over this limit by trying to extend these two 1.6TB LUNS.

    These may be similar posts up on VMware's site.

    1) http://communities.vmware.com/message/1323224 Not able to recognize Storage luns from ESX 3.5 server 2) http://communities.vmware.com/thread/71152 LUN not mounting.

    • 0