Home / server / Questions / 819288
In Process
inzanez
Asked: 2016-12-08 06:46:43 +0800 CST

SAN, ISCSI and Multipath (MLAG?)

  • 772

I'm a bit confused regarding the setup of a SAN with ISCSI and multipath. We are upgrading our network to 10Gbit, so there are 2 10Gbit switches that are configured with MLAG for normal ethernet traffic. LAG groups are made on the appriorate NICs, so for ethernet only.

Now there's a SAN with 2 controllers. Each controller houses 4 x 10Gbit NICs. So my initial plan was to use all 4 NICs on each controller to maximize throughput, as there are multiple servers connecting to the SAN.

The servers connecting house 2 dedicated 10Gbit NICs for SAN traffic. Each NIC connects to one of the switches. There's no LAG on those interfaces, they are configured with a static address in the SAN network.

  1. Is it OK for the 2 controllers of the SAN to be in the same subnet? So to have one main SAN network. Or do I need an independent segment for each controller?
  2. What about the 4 NICs of each SAN controller? Do I just bundle 2 NICs in a LACP group on each controller? Or is LAG not needed at all? I'm a bit confused what a reasonable configuration would be. Connect 2 NICs of each controller to one switch? Would I assign each NIC a dedicated address in the SAN network and use all the 8 addresses in total for multipathing?

Well, I'm not a storage expert as one can see,...:-)

So I got an update on question number 1. Assuming I'm going to use 2 NICs of each SAN controller (not all 4 to keep it simple for the time being), it would make sense to use two different subnets. It seems that this increases the chance that MPIO is being used efficiently.

If all the NICs are on the same subnet there's no guarantee that MPIO is being used.

I'm still investigating answer 2 though. I found multiple articles describing a setup as outlined above, but just using 2 NICs of each SAN controller. I tested that out, but contradicting to all the articles I don't get 1 single ISCSI target, I get two (one for each controller). When configuring MPIO using both targets my throughput drops from 1800MB/s to around 30MB/s,...

Switches: 2 x Mellanox SX1012

Storage: QSAN XS5200 with 4 x SFP+ 10Gbit adapter per controller

Server: Supermicro 2028TP-HC1R-SIOM, 4 x SFP+ Intel X710

The current configuration looks as follows:

enter image description here

storage-area-network iscsi multipath

2 Answers

  1. Since you have iSCSI SAN, I suggest you go with MPIO splitting iSCSI networks and utilize different subnets. LACP does same failover as MPIO, however comes with performance penalty due to unsupported Multiple Connections per Session (MCS) meaning, it would not performance boost for same configuration. https://www.starwindsoftware.com/blog/lacp-vs-mpio-on-windows-platform-which-one-is-better-in-terms-of-redundancy-and-speed-in-this-case-2

    Tweak the performance configuring NICs and switches Jumbo Frames value of 9k.

    • 4

  2. This is a case where you should follow your storage vendor's guidelines.

    I don't know who will be consuming your SAN resources, but the general approach should be:

    • Your regular servers using LACP should have LACP bonds comprised of links to both switches.
    • Your MLAG should have some sort of cross-switch peer link.
    • Your iSCSI should be delivered via MPIO, using one or more ports from one controller to one switch on a specific VLAN, and one or more ports from the other controller to the other switch on a different VLAN.
    • Those iSCSI VLANs should be unique to each switch and not participate in the MLAG.
    • If your servers will be using iSCSI directly, go with the MPIO approach... two VLANs and one port from the server to each switch. This may complicate normal data if you only have two ports per server.
    • 1