I am playing around with Ubuntu 18.04 and I'm noticing TCP performance regression that prevents me from upgrading my current running servers since they are very latency-sensitive.
For my particular use-case, I implemented a simple TCP test program that I'm running as a server, waiting for a certain number of clients to be connected before sending a batch of messages to all clients. I then measure the time it takes to sendmsg() a buffer of fixed-size to N clients.
Code sample
I'm running the server and the clients on two distinct machines, located in the same datacenter.
Results
Ubuntu 16.04
Version
srv01:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
srv01:~$ uname -ra
Linux srv01 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
sysctl
net.core.netdev_max_backlog = 3000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_moderate_rcvbuf = 0
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_rmem = 131072 1048576 16777216
net.ipv4.tcp_wmem = 131072 1048576 16777216
Run
srv01:~$ taskset -c 15,17,19 ./iouring --port 4040 --clients-count 3 --buffer-size 128 --batch-size 100 --sleep-ms 1 --total-messages 500000 --sockopt n
Send latency report
Min: 679ns
Mean: 1048ns
Max: 13949ns
p(0.100000) = 726ns
p(0.200000) = 734ns
p(0.500000) = 750ns
p(0.800000) = 859ns
p(0.900000) = 1338ns
p(0.990000) = 5024ns
Ubuntu 18.04
Version
srv02:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic
srv02:~$ uname -ra
Linux srv02 4.20.17-042017-generic #201903190933 SMP Tue Mar 19 13:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
sysctl
net.core.netdev_max_backlog = 3000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_moderate_rcvbuf = 0
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_rmem = 131072 1048576 16777216
net.ipv4.tcp_wmem = 131072 1048576 16777216
Run
srv02:~$ taskset -c 15,17,19 ./iouring --port 4040 --clients-count 3 --buffer-size 128 --batch-size 100 --sleep-ms 1 --total-messages 500000 --sockopt n
Send latency report
Min: 819ns
Mean: 4660ns
Max: 25061ns
p(0.100000) = 1379ns
p(0.200000) = 2660ns
p(0.500000) = 4871ns
p(0.800000) = 6559ns
p(0.900000) = 7416ns
p(0.990000) = 9444ns
As you can see, there is a significant performance drop on Ubuntu 18.04 with kernel 4.20 (I'm observing the exact same on kernel 4.15 and 4.18). I'm using the exact same machines (same hardware).
Somehow, it looks like it is related to TCP_NODELAY being enabled or not. When disabling TCP_NODELAY (no --sockopt), I'm getting a 50th time of 1005ns (Ubuntu 18.04).
srv02:~$ taskset -c 15,17,19 ./iouring --port 4040 --clients-count 3 --buffer-size 128 --batch-size 100 --sleep-ms 1 --total-messages 500000
Send latency report
Min: 817ns
Mean: 1790ns
Max: 15374ns
p(0.100000) = 955ns
p(0.200000) = 964ns
p(0.500000) = 1006ns
p(0.800000) = 1601ns
p(0.900000) = 4475ns
p(0.990000) = 9516ns
The same test on RHEL8, kernel 4.18 does not show any performance difference with or without TCP_NODELAY (950ns 50th).
Any idea what could cause this ? I can provide more details if needed.
Thanks