Questions[tcp-ip](ubuntu)

I'm seeing an issue with a specific combination of circumstances that causes TCP duplicate ACKs to be sent. This happens with a service hosted on Ubuntu Server 20.04.

I have a web service that, among many things, accepts uploads with multipart/form-data. When setting up a new version of the service on a new server I observed slow behavior and eventually landed on the following.

I have two computers to test from. When sending a request to it from a Windows 10 computer, some packets would be ACKed twice or ACKed out of order and then retransmitted. Here's a screenshot of Wireshark's TCP window scaling analysis when sending a 25 MB file with random bytes - the window never gets big enough to keep up throughput due to frequent (every 10-20 packets or so) duplicate ACKs or out-of-order ACKs and corresponding retransmissions. It takes around 55 seconds to transfer the file because of this.

When sending a request to it from a macOS 10.15 computer, I don't see this behavior. Here's a screenshot of Wireshark's TCP window scaling analysis - the window quickly grows and all the data is transfered in a handful of seconds the way it should be.

Using identical versions of Chrome and Firefox and also Safari on Mac and (old) Edge on Windows, the behavior is the same - all browsers on each computer are affected by the same (respective) behavior. This seems to me to indicate that there is some sort of TCP parameter tweak needing to happen.

To isolate the web stack I'm using, I wrote a minimal test program in Node.js v12 which accepts uploads with multipart/form-data, and I get identical behavior - bad behavior from Windows and good behavior from Mac. Node.js uses libuv and the other stack is ASP.NET Core using the Kestrel server which is configured with a non-libuv transport, so their networking have nothing in common above the OS's sockets.

As far as I can tell, both the good behavior and the bad behavior is maintained without modification when reverse proxied through nginx (adding TLS termination).

You might say that this is related to the network on my side. Both computers reside on the same wireless network behind the same router, and the Linux server is hosted in a different country, but I also see similar behavior when I try using computers not from within my network, or through cellular data.

As a final parameter, when hosting the test program or the web service on the Mac and sending a similar upload request from the Windows computer, I get the good, fast behavior. In other words, the Windows computer is not just throttled or riddled with packet loss compared to the Mac - it can work, but somehow when connected to the Linux server it ends up not working. (The Windows computer and Mac post similar scores in bandwidth tests.)

So, all of this seems to tell me that there's some sort of bad dice roll when Ubuntu 20.04's TCP settings (or the way they're configured at my host or affected by my host's environment) meets the Windows TCP stack's settings. What could cause these things, and what can I change on the server to hopefully force a good behavior?

(It's possible this is more of a generic Linux networking question - I'm posting it here because I'm running into it on Ubuntu Server 20.04.)

I am playing around with Ubuntu 18.04 and I'm noticing TCP performance regression that prevents me from upgrading my current running servers since they are very latency-sensitive.

For my particular use-case, I implemented a simple TCP test program that I'm running as a server, waiting for a certain number of clients to be connected before sending a batch of messages to all clients. I then measure the time it takes to sendmsg() a buffer of fixed-size to N clients.

Code sample

io_uring.c client.c

I'm running the server and the clients on two distinct machines, located in the same datacenter.

Results

Ubuntu 16.04

Version

srv01:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

srv01:~$ uname -ra
Linux srv01 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

sysctl

net.core.netdev_max_backlog = 3000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_moderate_rcvbuf = 0
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_rmem = 131072 1048576 16777216
net.ipv4.tcp_wmem = 131072 1048576 16777216

Run

srv01:~$ taskset -c 15,17,19 ./iouring --port 4040 --clients-count 3 --buffer-size 128 --batch-size 100 --sleep-ms 1 --total-messages 500000 --sockopt n
Send latency report
Min: 679ns
Mean: 1048ns
Max: 13949ns
p(0.100000) = 726ns
p(0.200000) = 734ns
p(0.500000) = 750ns
p(0.800000) = 859ns
p(0.900000) = 1338ns
p(0.990000) = 5024ns

Ubuntu 18.04

Version

srv02:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic

srv02:~$ uname -ra
Linux srv02 4.20.17-042017-generic #201903190933 SMP Tue Mar 19 13:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

sysctl

net.core.netdev_max_backlog = 3000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_moderate_rcvbuf = 0
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_rmem = 131072 1048576 16777216
net.ipv4.tcp_wmem = 131072 1048576 16777216

Run

srv02:~$ taskset -c 15,17,19 ./iouring --port 4040 --clients-count 3 --buffer-size 128 --batch-size 100 --sleep-ms 1 --total-messages 500000 --sockopt n
Send latency report
Min: 819ns
Mean: 4660ns
Max: 25061ns
p(0.100000) = 1379ns
p(0.200000) = 2660ns
p(0.500000) = 4871ns
p(0.800000) = 6559ns
p(0.900000) = 7416ns
p(0.990000) = 9444ns

As you can see, there is a significant performance drop on Ubuntu 18.04 with kernel 4.20 (I'm observing the exact same on kernel 4.15 and 4.18). I'm using the exact same machines (same hardware).

Somehow, it looks like it is related to TCP_NODELAY being enabled or not. When disabling TCP_NODELAY (no --sockopt), I'm getting a 50th time of 1005ns (Ubuntu 18.04).

srv02:~$ taskset -c 15,17,19 ./iouring --port 4040 --clients-count 3 --buffer-size 128 --batch-size 100 --sleep-ms 1 --total-messages 500000
Send latency report
Min: 817ns
Mean: 1790ns
Max: 15374ns
p(0.100000) = 955ns
p(0.200000) = 964ns
p(0.500000) = 1006ns
p(0.800000) = 1601ns
p(0.900000) = 4475ns
p(0.990000) = 9516ns

The same test on RHEL8, kernel 4.18 does not show any performance difference with or without TCP_NODELAY (950ns 50th).

Any idea what could cause this ? I can provide more details if needed.

Thanks

I'm running a type of perf test where I have a simple TCP server with 4 IP addresses that is listening on a port and getting connections from several other computers on the local network. Everything works fine up to just under 120,000 active connections, clients are able to get messages from client and create new connections. At just under 120,000, new connections just stop appearing. There is no log activity on server and clients start getting timeouts after a bit. There is no firewall that would be getting in the way. I have tweaked a bunch of settings already:

/etc/sysctl.conf

net.core.netdev_max_backlog = 1000000

net.core.netdev_budget = 50000
net.core.netdev_budget_usecs = 5000

net.core.somaxconn = 1024000

net.core.rmem_default = 1048576
net.core.rmem_max = 16777216

net.core.wmem_default = 1048576
net.core.wmem_max = 16777216

net.core.optmem_max = 65536

net.ipv4.tcp_rmem = 4096 1048576 2097152
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_max_syn_backlog = 3000000
net.ipv4.tcp_max_tw_buckets = 2000000

net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1

/etc/security/limits.conf

* soft nofile 6553600
* hard nofile 6553600

cat /proc/sys/fs/file-max
1621708

The limits are intentionally completely overkill because it's just a test. Are there some other settings I am missing that would enable more connections? Neither the CPU nor RAM is being stressed so I would like to keep pushing the hardware. Server and clients are all running on AWS EC2 t3a.xlarge instances, if that makes any difference.

Is it possible to redirect all calls to an external IP on internet, to localhost (without use of hosts)?

I have an old application and I don't have the source code and this application try to connect to a postgres server using an IP

Can I instruct ubuntu to redirect all callings to this external IP to localhost?

In this way, I'll be able to run a postgres server on localhost and work with my application.

note: the application uses the IP, not the dns.

TCP ACK is described here: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment

I know very little about this - I just learned about it via this ticket that I filed with Node.js:

https://github.com/nodejs/node/issues/21091

Basically I am using TCP to do IPC via localhost, but there's a big delay for each request.

This is the way to change the delay on RedHat apparently:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/tuning_guide/reducing_the_tcp_delayed_ack_timeout

is there a way to change this setting on Ubuntu?

Maybe like the wikipedia article says, use the TCP_QUICKACK option?