Questions[10gbethernet](server)

I would like to find out the best possible configuration/hardware to deliver 40Gbps from a single server in this question.

Situation

We have a video share proxy server that offloads peaks from slow storage servers behind it. All traffic is HTTP only. The server acts as a reverse proxy (files that are not cached on the server) and a webserver (files that are stored on local drives).

There are currently something like 100TB of files and growing on the backend storage servers.

The caching mechanism is implemented independently and this question is not about caching itself as it works very well - currently delivers 14Gbps, passes to the backend servers only 2Gbps. So the cache usage is good.

Goal

Achieve 40Gbps or even more throughput from a single machine.

Hardware 1

HW: Supermicro SC825, X11SSL-F, Xeon E3-1230v5 (4C/[email protected]), 16GB DDR4 RAM, 2x Supermicro 10G STGN-i1S (LACP L3+4)

SSD: 1x 512GB Samsung, 2x 500GB Samsung, 2x480GB Intel 535, 1x 240GB Intel S3500

System:

• irqbalancer stopped
• set_irq_affinity for each interface (via script in ixgbe driver tarball)
• ixgbe-4.3.15
• I/O scheduler deadline
• iptables empty (unloaded modules)
• Filesystem: XFS

Nginx:

• sendfile off
• aio threads
• directio 1M
• tcp_nopush on
• tcp_nodelay on

As seen on the graphs, we were able to push 12.5Gbps. Unfortunately the server was unresponsive.

There are 2 things that took my attention. The first one is high amount of IRQ. In this case I don't unfortunately have graphs from /proc/interrupts. The second thing was high system load, which I think was caused by kswapd0 having problems to work with 16G of RAM only.

Hardware 2

HW: Supermicro SC119TQ, X10DRW-i, 2x Xeon E5-2609v4 (8C/[email protected]), 128GB DDR4 RAM, 2x Supermicro 10G STGN-i1S

SSD, System configuration are the same as hardware 1. Nginx is sendfile on (aio/sendfile compared further).

This seems better, so now as we a have server, which works in peaks, we can try some optimizations.

Sendfile vs aio threads

I tried to disable sendfile and use aio threads instead.

• sendfile off
• aio threads
• directio 1M (which matches all files we have)

vs

• sendfile on

Then at 15:00 I switched back to sendfile and reloaded nginx (so it took a while to finish existing connections). It is nice that the drive utilization (measured by iostat) went down. Nothing has changed on the traffic (unfortunately zabbix decided not to collect the data from bond0).

sendfile on/off

Just tried to switch send on/off. Nothing has changed except Rescheduling interrupts.

irqbalancer as a server/cron/disabled

As @lsd mentioned I tried to setup irqbalancer to be executed via cron:

*/5 * * * *   root    /usr/sbin/irqbalance --oneshot --debug 3 > /dev/null

Unfortunately it didn't help in my case. One of the network cards started behaving strange:

I couldn't find what was wrong in graphs and as it happened the next day again, I logged in to the server and saw that one core was at 100% (system usage).

I tried to start irqbalance as a service, the result was still the same.

Then I decided to use the set_irq_affinity script and it fixed the problem immediately and server pushed 17Gbps again.

Hardware 3

We did upgrade to new hardware: 2U 24 (+2) drives chassis (6xSFF), 2x Xeon E5-2620v4, 64GB DDR4 RAM (4x16GB modules), 13x SSD, 2x Supermicro (with Intel chip) network cards. New CPUs improved the performance a lot.

Current setup remains - sendfile, etc. Only difference is that we let only a single CPU handle both network cards (via set_irq_affinity script).

20Gbps limit has been reached.

Next goal? 30Gbps.

Feel free to shoot at me ideas how to improve the performance. I will be happy to test it live and share some heavy graphs in here.

Any ideas how to deal with large amount of SoftIRQs on the cpu?

This is not a question about capacity planning - I already have the hardware and the traffic. I can always split the traffic to several servers (which I will have to do in the future anyway) and fix the problem with money. This is however a question about system optimization and performance tunning in a real live scenario.

Can anyone tell me what command i run to determine if my 10G NIC is running in single RX-TX queue mode or multiqueue? It looks like it only has 1 RX/TX queue according to cat /proc/interrupts

root@hostname:scripts]# cat /proc/interrupts | grep ens1f0
  94:  360389979          0          0          0        184          0        330          0          0          0          0          0          0          0          0          0          0        169          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ens1f0-TxRx-0
  95:          4          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ens1f0

If it is operating in single queue mode, how would I enable multiqueue ?

My question is what's is the difference between Gigabit Ethernet Controller and Ethernet Converged Network Adapter?

I want to upgrade my server to 10 Gigabit ethernet, my server vendor recommend me one card but it's very expensive and there are other cheaper ethernet card and since they are all connected with PCI to the board I wanted to know can I use any other PCI ethernet card?

The vendor recommended me to purchase Intel 82599 10GbE Daughter Card, Gen 2 PCIe x8 and I was thinking to buy the Intel® Ethernet Converged Network Adapter X540-T2, I didnt wanted to put useless information but if you need any additional information to answer this question I will happy to share it with you.

I am building home cluster where I going to have about 16 nodes which can live with 1G ports, but I want to have 10GE on file server & central node. It's all local, so no need for cables longer than 3-5m. And of course I want to spend as little money as possible (not going to spend more than whole cluster costs) :-)

What are my options?

1. Legacy solution is to take some 24-48 port 1GE switch, and connect to file/central nodes via 4-8 aggregated links. This will work I guess, cost is very acceptable, but I am not sure if it's ok to use that much aggregated links. And of course it would be hard to double bandwidth when needed... :-D
2. Switch with several 10GE uplink 'ports'. As far as I see, they all require modules which costs about 1000$, so I will need 4 10G modules, and 2 10GE cards... Smells like way more than 5000$+...
3. Connect file & central node via 2 10G cards directly, and put 4 quad-port 1GE NICs on fileserver. I am saving on 2 10G modules and a switch, fileserver will have to do packet routing, but it's still gonna have a lot of CPU's left :-)
4. Any other options? Infiniband?
5. Are MyriNet adaptors works fine? I guess there are no cheaper options?
6. Hmm... Scrap fileserver, put it all on central node and provide dedicated 1GE port for each of the nodes... This is sad...

We're speccing up some 10GbE switches for integrating a few older servers into our Equallogic SAN, and we're noticing quite a price gap between SFP+ and Copper (Cat 6A) equipment (Dell 8024F vs 8024).

I'm not really sure what the real-world difference is between the form factors. The Dell guys tell me that SFP+ has a lower latency, but couldn't tell me much more than that besides that our M1000e and PS6010XV chassis's only comes with SFP+ uplinks (and SFP+ is substantially cheaper).