Home / server / Questions / 278072
Accepted
BrandonB
Asked: 2011-06-08 15:32:27 +0800 CST

Design Critique Needed ( VLAN & ESXi )

  • 772

So this is my new "project" which consists of tearing down our legacy network and re-doing it correctly. As of now it is a mess. Production is a hogpoge of legacy servers that should have been put to pasture two years ago. Within this time period I got my VCP, which is good but I do not know all of this by heart. This is my first network / server revamp. As well as my first non-lab vSphere work, actually I have never logged into a SAN up until a few weeks ago when we received our PS4000 At this point am breaking this project into design phases so the first one up is the VLAN, Switching layout. I have attached my spread sheet with vlan id's, subnets, use and switch ports. I have read allot of best practices but, it is starting to just melt together. If i could get any advice on this I would very much appreciate and, please feel free to rip it apart.

VLAN Assignments Rev.1

http://dl.dropbox.com/u/6674806/Copy%20of%20Production%20VLAN%20Port%20Assignments-1.xlsx

enter image description here

Kind Regards,

Brandon

VLAN Assignments Rev. 2

Updated layout

Uploaded a updated version per advice.

networking vlan storage-area-network vmware-esxi infrastructure

3 Answers

  1. Best Answer

    You have done this absolutely correctly! It appears to me also that you're using Juniper switching based on the format of your interfaces.

    Recommendations:

    Make sure you set MTU of 9216 on the Juniper switch for any port carrying iSCSI. Equallogic likes jumbo frames. You should set ALL the links from the switch to your ESXi systems as trunks, carrying the appropriate VLANs, this gives you flexibility.

    Finally, install the Dell flavored version of ESXi so it has the open manage and systems health plugins, and afterwards install the Dell multi-pathing plug-in for ESXi and let it create your iSCSI vswitch for you, this makes sure that the vswitch, and in turn each vmnick and vmkernel port all are set correctly for jumbo frames.

    Remember also how vmware blocksizes affect the datastores on your SAN, typically unless there is a requirement we create a datastore with a 1MB block size, but this limits you to a VMDK of like 512GB or something like that I can't remember, a 2MB block size will let you do 2TB VM's, and so on. (I can't remember the figures).

    • 4

  2. @SpacemanSpiff has put some good recommendations out there, a couple things to add:

    1. I don't see any notes on which interface the VMware HA communication will be using. By default it'll probably grab the management interface, which should be fine; it's not a lot of traffic. But make sure you're aware of which interface it's using; if, for instance, you added a host on a different vlan for management but the same VMotion and FT vlans, then the HA cluster wouldn't accept that host.

    2. Not sure what kind of physical NICs you've got, but I'd recommend making sure a given vSwitch has uplinks using multiple physical NICs, so a card failure can't bring you down. So, for instance if you've got 2x 4-port NICs, then instead of doing management with vmnic1 and vmnic2, do it on 1 and 5.

    • 4

  3. The overall approach is sound. Some comments:

    • You may have too many adapters and at the same time too little bandwidth. Specifically, there's no reason to have 2 pNICs for management and 2 pNICs for vMotion. Use 2 pNICs for both, but configure one to be active for management and standby for vMotion, and the other active for vMotion and standby for management. As for bandwidth, FT over gigabit may not work. You need to calculate the needs for FT using the formulas in http://www.vmware.com/files/pdf/perf-vsphere-fault_tolerance.pdf.
    • For iSCSI, you must use Round Robin or EQL MEM extension, not just any multipathing.
    • Use iSCSI-enabled Broadcom NICs and configure hardware offload per http://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf. Keep in mind that Broadcoms are not real expensive iSCSI HBAs, and they don't support jumbo frames. Still, you'll have better performance than with pure software iSCSI and jumbo frames.
    • You list various VMs that will use different VLANs, but you only specify a single VLAN for your "data" (VM network, I assume?) pNICs. Instead, they should be configured as trunks, with multiple portgroups on the VM side.
    • Look at your budget numbers and decide if you want to bump up the license level to Enterprise. That will give you DRS, Storage vMotion, and VAAI. It is also a prerequisite for EQL MEM. Since you're building a three server cluster, your best purchase is one of the acceleration kits, and the difference between Adv and Mid (w/ Ent. ESXi) is ~$5K.
    • Buy Dell Management Plug-in for VMware. This will be your favorite addon after MEM.
    • Don't skimp on memory, especially if your VMs are mostly Windows. Pack those PowerEdges with as much RAM as your budget allows.

    I'm sure I could think of more, but this should be helpful too.

    • 3