Home / server / Questions / 1161977
Accepted
pgoetz
Asked: 2024-07-04 04:45:09 +0800 CST

Poor network bandwidth on a point to point connection between 2 linux machines with Mellanox ConnectX-6 cards

  • 772

For both machines:

  • Ubuntu 20.04
  • Mellanox MT28908 ConnectX-6 cards in ethernet mode
  • Kernel module: mlx5_core (from MLNX_OFED_LINUX-24.04-0.6.6.0-ubuntu20.04-x86_64)

The ConnectX-6 cards are directly connected to each other using an Nvidia DAC using RFC1918 addresses; i.e. a little 2-node network. The machines are server grade rackmount systems, with one having an Intel Xeon Gold 5318Y CPU, and the other 2 x Intel Xeon Platinum 8358 CPUs. These cards are supposed to provide 100Gb/s connectivity. However

root@zakynthos:~# iperf3 -c 192.168.101.1
Connecting to host 192.168.101.1, port 5201
[  5] local 192.168.101.2 port 45974 connected to 192.168.101.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.17 GBytes  27.2 Gbits/sec  227    912 KBytes       
[  5]   1.00-2.00   sec  2.58 GBytes  22.1 Gbits/sec  1382    755 KBytes       
[  5]   2.00-3.00   sec  2.36 GBytes  20.3 Gbits/sec  2459   1.08 MBytes       
[  5]   3.00-4.00   sec  3.07 GBytes  26.4 Gbits/sec  3433    723 KBytes       
[  5]   4.00-5.00   sec  3.36 GBytes  28.9 Gbits/sec  3004   1.05 MBytes       
[  5]   5.00-6.00   sec  2.83 GBytes  24.3 Gbits/sec  1692    942 KBytes       
[  5]   6.00-7.00   sec  2.47 GBytes  21.2 Gbits/sec  1819   1022 KBytes       
[  5]   7.00-8.00   sec  3.03 GBytes  26.0 Gbits/sec  1578    841 KBytes       
[  5]   8.00-9.00   sec  3.37 GBytes  28.9 Gbits/sec  1920   1024 KBytes       
[  5]   9.00-10.00  sec  2.91 GBytes  25.0 Gbits/sec  1239   1.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  29.1 GBytes  25.0 Gbits/sec  18753             sender
[  5]   0.00-10.04  sec  29.1 GBytes  24.9 Gbits/sec                  receiver

I'm used to seeing iperf3 deliver around 90% of the rated bandwidth for a card, so to me this looks terrible and unacceptable. Anyone have any insights as to why I'm not seeing at least 80-90Gb/s performance here?

linux

2 Answers

  1. server grade rackmount systems - what are those systems and specifically which slot have you plugged the NIC in?

    The MT28908 ConnectX-6 supports PCIe 4.0 x16 with a theoretical bandwidth of up to 252 Gbit/s - sufficient for 2x 100 GbE. However, if that slot only supports e.g. PCIe 3.0 x4 electrically ("x16 @ x4"), then bandwidth drops to 31 Gbit/s with pretty much the result you're seeing.

    Another possibility is a poor memory setup with just a single channel (admittedly unlikely).

    Check your server specs.

    • 2
  2. Best Answer

    The problem was the version of iperf3 that ships with Ubuntu 20.04 is incapable of handling bandwidths greater than around 25Gb/s. I downloaded and compiled the newest version of iperf3 (which supports running parallel streams), and running 16 parallel client streams:

    /opt/iperf-3.17.1/bin/iperf3 -c 10.119.8.9 -P 16

    I was able to get around 98-99Gb/s, which is close enough to the theoretical max for the card.

    • 2