How to improve Intel X520-DA2 10Gb NIC throughput without Jumbo packets

One of the problems with high performance NIC's is that the modern PC architecture has a bit of trouble keeping up. But, in your case, this isn't so much the problem. Let me explain.

The CPU has to do a lot of work processing TCP packets. This affects the throughput. What's limiting things in your case is not the network hardware, but the ability of the server to saturate the network links.

In more recent times, we've seen processing move from the CPU to the NIC like checksum offload. Intel have also added features to help reduce the load further. That's cool and I'm sure all optimizing features are turned on.

As you've alluded to, jumbo frames - actually that helps throughput somewhat. But not as much as RDMA.

Most 10GBit ethernet hardware will have a very nice underutilized feature called RDMA or remote direct memory access. It allows the NIC to do memory to memory copies over the network without the intervention of the CPU. Well, OK the CPU tells the NIC what to do and then the NIC does the rest. The trouble is, it's not used much yet. But it's getting there. Apparently, in the most recent version of Microsoft Windows Server 2012, they have something called SMB Direct. It uses RDMA. So, if you want to increase throughput, you want to use that.

Are you able to put together some test hardware and install it onto there to see how it performs?

By the way, I'm not sure if you will see it at 10Gbit so much, but fast RAM helps with RDMA especially with 56Gbit Infiniband. In general, it's best to use the fastest RAM your server supports.

Also note this comment on the SMB Direct link I put above:

You should not team RDMA-capable network adapters if you intend to use the RDMA capability of the network adapters. When teamed, the network adapters will not support RDMA.

Update: Looks like not ALL 10GBit NIC's support RDMA for some reason. So check your model's features first.

Another thought I had was the type of protocol being used to do your testing may be affecting the results. i.e. protocol overhead on top of TCP overhead. I suggest you look into using something that can test without touching the hard drive such as iperf. There is a windows port of it somewhere.

I think this question: Why does my gigabit bond not deliver at least 150 MB/s throughput? is related to your problem. I was talking about a Dell PowerEdge 6950 there. The answer was basically "use jumbo frames" to reduce the interrupts. I can imagine that tuning the offload-engine of the network-card might help in your case, but I do not know how to do so on W2K8R2.

Idea: Raise number of buffers in the network card, raise the interrupt-triggor for packets in the buffer, so each interrupt will handle more packets (i.e pass them to the OS-IP-stack).

See this link: Setting coalescence-parameters with ethtool for 10 Gb this is what I am basically referring to.

Your cpu utilization screenshot shows 2 potential bottlenecks:

1. 4 cores maxing out doing kernel work (i.e. probably interrupt handlers processing packets)
2. 1 core maxing out in - mainly - user mode

To address the former:

• Try changing Interrupt moderation settings, depending on your drivers it is more than just on/off, you may be able to set a moderation strategy
• Try disabling/enabling all the offload features (in your case, disabling might be beneficial, so as to move a potential bottleneck from your (single-core) NIC, which the functionality would be offloaded to, to your (multi-core) processors)
• Try enabling "Receive Coalescing" (when receiving TCP), and the various "Large Receive ...", "Large Transmit ..." etc features your driver might provide
• Can't you set your RSS queues to a value higher than 4? It seems only one of your 2 ports is being used (as you said you are aware of it, I assume you set your second port to at least 4 (or 8, not sure if HT has to be counted)
• If possible, increase the number of different TCP/UDP ports used, or IP source/target addresses, because one address/port/protocol 5-tuple (or address/protocol 3-tuple for non-TCP/UDP traffic) will always have to go to the same core no matter what your RSS settings

As to the latter (not knowing what app you are actually using):

If that 1 core maxing out in user mode indicates your single-threaded (or single-thread bottle-necked) app, it should be

• fixed, or
• reconfigured (e.g. increase # worker threads if possible), or
• redesigned

to use multiple cores, which might or might not be trivial.

Also, as your app (if it is indeed your app), apparently runs on a NUMA node #1, but packet handling by the kernel is done on NUMA node #0,

• try affinitizing the app to NUMA node #0

E.g. by right clicking on the process in Task Manager, which will give you the option to change that, at least in Win2012R2. I tried it, and for me it did not help, but it's worth a try, as it might improve the cache hit rate.

Btw, is the machine in question sending? Receiving? Both? In terms of configuring your system for performance, sending and receiving are almost completely unrelated, although my suggestions above cover both.