john01dav's questions

I recently asked a similar question about how to transfer large (many gigabyte to many tens of gigaytes) files over the Internet when latency is high. I got a great answer, but only for downloads. When I want to upload something, from my computer that is behind a firewall that only allows outgoing TCP connections, to a server, I can't get lftp or any other tool to satuate my gigabit upload (to a gigabit server that is 115ms away). lftp seems to cap out around 36.88mbps, even when doing a parallel transfer, such as with this command:

lftp -e 'mirror --parallel=1 --use-pget-n=10 -R . /destination/path' sftp://user@destinationdomain

I also tried some variations of this command (e.g. using put instead of mirror, and different flags), but nothing gets anywhere near saturating my gigabit upload. It's possible in this case that there's a bad link between me and the server, since although both are gigabit neither is a particularly good network (both have issues when sending data to far-away places, sometimes). To test this hypothesis, I ran a speedtest.net speed test to another ISP in the same city as the destination server (the host of the server does not host a speedtest.net server), and I got this result:

which seems to rule out any issue with intermediate networks (at least, anything severe enough to cause 36.88mbps), which is further strengthened by the fact that not only does my route to this speedtest.net server use the same long-distance transit provider (Hurricane Electric), but it even seems to go through the same datacenter that my server is hosted in with the same provider!

So, I ask, how can I easily (meaning with a single command) upload from my computer to a remote server while saturating my gigabit connection, or, at least, a significant fraction of it as speedtest.net is able to do?

Due to some really buggy software that likes to write its own backup files to its working area even when that it disabled, duplicity has backed up those backups resulting in several hundred GBs of useless data in its archives (I use duplicity to automatically backup the working directory of this software each day, which, when the software doesn't listen when it is told to not also backup, results in backups being backed up, which is an explosion of disk space on the order of O(n²) or so.). All of the problematic files are in a single directory, /path/to/software's/working/directory/backups, and I need to get these backups out of duplicity's archives so I have some breathing room on my server's SSD before a month goes by and the backups expire. How can I do this, without affecting the rest of the actually useful data that duplicity is storing?

I am currently dealing with a networking problem on high latency (100-400ms) Internet links. I run a Minecraft network, and I recently moved it to a separate datacenter to get a server with a better CPU and more ram. The users of this server are spread all over the world. Before the switch, the server was in Montreal and the users in Europe got ~100-200ms latency, and those in Australia got ~200-300ms latency. After the switch, the server is in Germany, the users in North America get ~100-200ms latency, and the users in South America and Australia get 200-400ms latency. Overall, the latency is quite similar, but who gets great latency and who gets tolerable latency varies (note that Minecraft isn't very latency sensitive in general, especially compared to most video games). There also isn't any significant packet loss, as measured by the MTR and ping tools. Furthermore, the software on both servers is almost identical. Both servers run Debian 10, and I tar'd all software + configuration that's not in APT repositories to send that over, while reinstalling the exact same packages via apt. As such, the software configuration should be essentially identical.

Yet, many users are having connection trouble. It seems to only occur around 6:00 PM (± a few hours) in the eastern United States. The connection trouble specifically takes the nature of the throughput of all TCP connections being atrociously low. With an SSH+SOCKS proxy, it took minutes to load a normal web page (Gmail), and in-game, it often takes minutes for even a simple chat message to get through if a few MBs of world data are currently being transferred. The effective latency of a TCP connection (e.g. the time it takes a chat message to get through it) increases unreasonably massively when any data is put over that TCP connection. A normal SSH session with just a terminal is basically fine, and the game is mostly fine if not much is going on, but as soon as anything of significant size gets sent over TCP and it is during the aforementioned time, then the throughput breaks down and latency over TCP (but not via ping) becomes unreasonable, even to multiple minutes in the worst cases. When this issue first took place, there was significant packet loss (~25%) which I thought was to blame, but, that packet loss is no longer occurring (according to ping, etc.) yet the issue remains. The packet loss, but not the problematic symptoms, disappeared after I made a general report to the new host about packet loss, but before I could give them more detailed data with MTR as they requested in their response to that report. My impression from the host is that they did not change anything, but, who knows really.

As such, at this point, I suspect that the relevant difference between the servers is that the old host (OVH) does some sort of tuning of their OS images (something that I know to be the case), and that the new host (Hetnzer) does not.

I suspect that this tuning has something to do with TCP window size, but when I attempted to manipulate those settings to make changes, the settings didn't seem to do what they're supposed to. Specifically, when I set the various net.ipv4.mem or net.core.mem settings that I find listed on the Internet via sysctl, the window size that iperf selects (or the maximum that it is allowed to select, when using the -w option) seems to take a random value with no discernible-to-me relation to the values that I set via sysctl, as opposed to behaving in the way that I expect, where it's maximum value is simply whatever I set via sysctl. Note that iperf -s misbehaves even before a client connects to it, so failing to make the same changes on the client is not a plausible explanation.

As such, I'm wondering 2 things:

1) How can I fix my server, and allow the latency in TCP connections to be similar to the actual latency on the link, even at peak times and under moderate load (a few mbps)?

2) How can I reliably and predictably change the TCP window size of all applications? (or, equivalently, what's going on with the sysctl settings applying in seemingly random ways? / what is the pattern that I'm missing?)

A few times per year, I need to transfer massive files (20+GBs) over the Internet (latency 20-100ms), but with reasonably fast connections on the order of 1gbps between servers. Unfortunately, when I use rsync to do so it has all sorts of issues and repeatedly fails to saturate the connection, only using all of it for a fraction of a second at a time and often using none (< 1mbps, possibly zero) for minutes on end. Neither my hard drive nor CPU on either end are pinned at 100% usage, so neither of them can be the bottleneck. SCP behaves in the same way.

Speed tests, however, manage to fully saturate my connection for the duration of the test (much longer than rsync can). I can get some success by using HTTP over Cloudflare with a multithreaded downloader (aria2c), but it still won't saturate my connection — it simply prevents it from going near zero. Interestingly, aria2c also won't saturate my connection when not going through Cloudflare.

In my research before asking this question, it seems like TCP window may have something to do with this, but I don't have a proper understand of what that is nor do I know how to change it (the only information that I can find on the Internet is in the context of speed tests, where the speed test has a command line flag, but neither rsync nor SCP have any such flags documented in their man pages). This isn't useful, however, as I have no idea how to use this knowledge to fix my download speed.

As such, I ask, how can I easily (e.g. without mucking around with custom compilation, or paying for proprietary products), like I can with rsync, scp, or http (via nginx) now, saturate my fast connections for bulk file transfers? An ideal answer will give me a set of rsync flags that I can use, in order to fully solve this problem. A perhaps more realistic answer that would still be useful would be a particular standard Linux utility that can accomplish what I want without significant tweaking. I am sure that such a thing exists, as I am sure that I am not the only one who needs to transfer large files over the Internet.

I currently rent a dedicated server from OVH, and I was looking at nethogs to see how much connection capacity a specific process was using. I ended up, however, finding a slew of processes there that I did not authorize that are communicating with IPs all over the world (so far the list includes China, Brazil, US (multiple states), Sweden, United Kingdom, and the Netherlands), based on their names. The full rows in the table for these processes take the form ? root <my server's ip>-<some other ip>. Running nethogs as root does not change this. Using netstat to try to figure out the PID of these results in it saying the PID/Command is equal to -. Some frantic googling, after thinking my server has been hacked, gave me the idea that these are kernel modules using the network in much the same way that NFS does. Looking in lsmod, I see a large number of legitimate-sounding names that I don't recognize, so that is not useful. Even so, a rogue module could call itself something else. As such, I would like to ask how I can tie these connections to specific kernel modules, and then do further research to figure out what is going on.

Thank you

Recently I saw the whois record for google.com, and it has none of the usual information such as the admin's contact details. It is extremely truncated:

Domain Name: GOOGLE.COM
Registry Domain ID: 2138514_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2011-07-20T16:55:31Z
Creation Date: 1997-09-15T04:00:00Z
Registry Expiry Date: 2020-09-14T04:00:00Z
Registrar: MarkMonitor Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: [email protected]
Registrar Abuse Contact Phone: +1.2083895740
Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited
Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited
Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited
Name Server: NS1.GOOGLE.COM
Name Server: NS2.GOOGLE.COM
Name Server: NS3.GOOGLE.COM
Name Server: NS4.GOOGLE.COM
DNSSEC: unsigned

Several other domains such as duolingo.com and even stackexchange.com are the same way. Why are these domains allowed to not have whois information? Is this something that anyone can access, for privacy protection?

According to this post with benchmarks, IIS consistently outperforms nginx in speed. Specifically, IIS uses 2.3x less CPU power than nginx, processes more than double the requests, and responds in less than half the time.

The blog post says nothing more about what IIS does internally to make it so fast. It simply states:

IIS, when used in conjunction with a Windows OS is fast and reliable.

I would like to know what exactly IIS (or perhaps Windows NT) does to make it so much faster than nginx on Linux. I was unable to find any other information on exactly what makes IIS so fast. I did, however, find another benchmark that agrees with the first one.