Home / server / Questions / 568167
In Process
Jeff Sacksteder
Asked: 2014-01-18 19:11:58 +0800 CST

Can SMB3 Multipathing be weighted by policy?

  • 772

Consider a configuration with two identical WS2012R2 servers. Both have quad-port network boards. Each has one NIC connected to a subnet shared by other servers and ultimately accessible to client machines. The remaining three ports on each server are 'nic teamed' and connected to a different subnet shared only by the two servers in question. What I would like to see is SMB traffic preferentially going over the faster 3-nic path and only resorting to the more congested 1-nic path if necessary.

Can I trust the load-balancing in SMB3 to do the right thing here? If not, can I apply some sort of weighting similar to tcp/ip route costing?

windows-server-2012

1 Answers

  1. I'm feeling fairly comfortable telling you that the product will "do the right thing" but I don't have a representative environment to simulate exactly what you're talking about.

    To talk about it technically, the devil is in details after the server computers communication details of their network interfaces via the FSCTL_QUERY_NETWORK_INTERFACE_INFO API. This negotiation is actually shown as an example in section 4.8 of the [MS-SMB2]: Server Message Block (SMB) Protocol Versions 2 and 3 specification. Unfortunately, the portion of the specification that would contain the logic associated with network connection selection just says "The client selects any one network interface pair to establish a new connection ...". There is some nuance in the interface selection protocol (which, helpfully, Microsoft has omitted from the specification, though I suppose one could argue that this behavior is outside the specification). This article, for example, describes how "non-routable" (read: APIPA and link-local addresses) are not used for SMB multichannel.

    Aside from the "non-routeable" addresses described in the article above, this really makes me wonder if there's an implicit assumption in the product that arbitrary network interfaces on a client/server pair are capable of communication. (This could well be a vein of fail just waiting to be mined in some oddball scenarios where traffic filtering / policy makes that assumption untrue.) I'm going to guess there's probably some preference for interfaces in the same subnet as the interface the initial channel was created on. (It would sure be nice if Microsoft published this behavior!) (I see that I'm not the only one who has been thinking about this, at least...)

    You can do a Get-SmbMultichannelConnection from the machine to see if SMB multichannel connection between the servers is being created during a long-running copy operation. Add the -IncludeNotSelected argument if you want to see the "not-selected" paths.

    You can out-right exclude connections from SMB multichannel with the New-SmbMultichannelConstraint cmdlet, but that's not weighting the channel priority, per se-- it's just exclusion (which isn't what you're looking for).

    As an aside: I assume you're using the built-in NIC teaming functionality in W2K12. You may want to check, with your switches, that the LACP hashing algorithm (called the "Load balancing mode" in Windows Server) chosen is optimal. I've seen some talk of changing that mode offering higher throughput with different switches.

    • 3