Home / server / Questions / 184465
Accepted
icky3000
Asked: 2010-09-25 12:35:09 +0800 CST

enabling jumbo frames on an iSCSI network after the fact

  • 772

We're using iSCSI storage and have two dedicated VLANs for iSCSi. We didn't implement jumbo frames initially. I'd like to turn it on now. I understand I need to turn it on for the NICs that connect to the iSCSI VLANs and on the switch ports they connect to and then do the same for the SAN itself. My question is about the timing of all of this. I've got 4 iSCSI boxes and 30 servers connecting to them. Can I make the change at different times without causing big trouble? For instance, if I go through and set all the NICs to jumbo frames first and then do the switches and then the storage, will I have issues if iSCSI traffic is moving at the same time? For obvious reasons, I'd prefer not to shut down all iSCSi traffic first. I think I can reasonably coordinate this work with the network guys to do it all in one evening and plan to enable flow control on the switch ports at the same time. Advice?

jumboframes iscsi

2 Answers

  1. Best Answer

    I would go this way:

    1) switches 2) storage 3) clients

    I've had several times in my practice that using iSCSI with 9k jumbos thru a switch with disabled jumbo could cause a slow as hell throughput. So, obviously, switches always go first, you actually don't change anything with this, you just ALLOW clients to use frames more than 1.5k but they of course can continue using a standard size frames w/o any restrictions.

    The second one will be a storage because (I'm not sure here) when the client initiates a tcp connection, it asks a storage to use a standard frame from start, if this client isn't switched yet to jumbo. So the storage can deal simultaneously with a clients using a jumbo frames aswell as with clients which're still on the standard frames.

    • 5

  2. I compare it to the flow of water. Upgrade the pipes before turning on the tap. In other words, enable jumbo frames on all the switches first, then the endpoints, which are the NICs and the SAN. I'm not sure in which order you should enable the endpoints.

    From the EqualLogic Network Performance Guidelines:

    To take advantage of Jumbo Frames, all devices in the network path between servers and the PS Series group⎯including the switches and the NICs used to access volumes⎯must have Jumbo Frames enabled.

    Switches configured for Jumbo Frames will support both standard Ethernet frames and Jumbo Frames. However, if a NIC is configured for Jumbo Frames, but the switch is not, you may experience inconsistent behavior. The switch will function properly if the frames are small, but once the NIC attempts to send frames larger than 1500 bytes, the switch will not be able to handle the frames and will drop them. Also, if some switches are configured for Jumbo Frames, but others are not, you may experience inconsistent behavior if routing changes occur after the connection has been established.

    In the case of our EqualLogic SAN, the new MTU size only takes effect for new iSCSI sessions, so force a failover or reset the sessions another way.

    When an iSCSI session is set up the EqualLogic performs Path MTU Discovery to determine the max. MTU size the entire path supports. If the path supports 9000-byte sized frames, they are enabled and this is logged in the SAN. Not sure how other SANs do this, so check your docs.

    • 3