Since at least a few months, issuing a netstat -t command on our web server, which has TCP ports 22, 80 and 443 exposed to the Internet, often reveals dozens of connections in SYN_RECV status:
$ netstat -nt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 128.1XX.XX.X:22 142.93.230.186:50603 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 104.248.89.196:36332 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 167.71.79.217:40430 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 142.93.235.90:36742 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 178.128.243.146:35451 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 178.62.253.100:43994 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 188.166.91.16:42461 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 178.62.242.138:33381 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 142.93.235.90:57784 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 165.22.198.207:36181 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 68.183.11.83:63899 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 104.248.93.253:41816 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 104.248.85.129:48833 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 178.62.207.43:59050 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 128.199.55.152:55891 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 178.62.205.23:52978 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 165.22.198.135:36668 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 165.22.203.65:65273 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 68.183.15.91:46722 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 167.71.4.136:55194 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 142.93.228.103:60458 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 178.62.253.100:52599 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 104.248.89.56:46283 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 178.62.250.235:43637 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 167.71.7.181:37556 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 104.248.89.200:64128 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 142.93.139.213:38821 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 104.248.92.161:63004 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 68.183.15.91:46210 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 104.248.92.138:39651 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 178.62.241.74:43953 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 142.93.224.74:42996 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 165.22.196.167:38597 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 178.128.254.3:36751 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 104.248.93.27:32904 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 167.71.78.255:60529 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 165.22.201.209:45397 SYN_RECV
tcp 0 0 128.1XX.XX.X:80 178.62.207.205:39291 SYN_RECV
tcp 0 0 128.1XX.XX.X:22 165.22.198.207:61967 SYN_RECV
tcp 0 0 128.1XX.XX.X:443 178.62.234.81:39309 SYN_RECV
[---cut---]
The first though I had was that it was a SYN flooding attack, however (a) the server doesn't seem to suffer at all from the "attack", (b) the SYN rate is pretty low, at just slightly over 2 packets/sec., and (c) I generally observe those SYN packets coming from the same netblock both on that server (University lab web server hosted in our University network) and on a home server which is on a completely different netblock (connected via a regular residential fiber connection with a public dynamic IP), and (d) the Linux kernel never complies about a possible SYN flood (TCP SYN cookies are enabled).
What I'm wondering, is what the actual purpose of those SYN packets can be. They don't seem to come from network problems (badly configured firewalls or similar issues). They almost always come from IP spaces of datacenters, which change from one day to another (it could be Amazon EC2, Serverius, DigitalOcean, etc.). Sometimes it's just one to a few IPs, sometimes there are dozens. Sometimes the IP answers if I ping it, sometimes not. Although I only tested sporadically, I've almost never found a reachable web server at those IPs on port 80 or 443 (never tried to scan more ports than this), but see below for the first one I found, today. They come by bursts for perhaps one hour, then disappear; I didn't try yet to automatically detect them to have a hourly plot to see if there is some "schedule".
I was also thinking that this might be some kind of amplification attack using forged IP source addresses (each received SYN packet will eventually produce 5 SYN ACK packets before the server times out), but it doesn't make much sense as (a) a normal server would quickly reply with a RST packet thus stopping further SYN ACKs, and (b) as mentioned, those servers don't appear to host public websites that might worth of a DDoS amplification attack.
The (possibly spoofed?) source addresses mostly have no PTR record, or a generic one, but in some cases they give actual hostnames (e.g. in the list mentioned above, 142.93.230.186 gives parimatchklub.com, apparently a Russian sports betting site). The site was very slow to reach, so perhaps it is indeed some form of reflection attack against the apparent source addresses?
Did anybody else already observe this? Any explanation about the actual purpose of those packets?
It looks like the footprint left behind by something like nmap host discovery. Nmap is a host discovery tool and port scanner.
nmap’s host discovery phase can send syn packets to webserver ports in order to detect servers which block icmp ping. Someone may be using nmap to discover live hosts in your ip range (or even 0.0.0.0/0) and may have no interest whatsoever in your host.
You could probably detect this by packet sniffing SYN and ICMP packets and comparing them with the scans described in: https://nmap.org/book/man-host-discovery.html
You can also see if they’re following up with a wider port scan. They may be looking for hosts with a particular port with a vulnerability known to them.
It could be HTTP Evasion to bypass your server security. The IP addresses are possible malicious hosts, because they are budget hosting providers, which are known to host Malware and participate in Botnet activity.
The output of
netstatshows that your server was unable toESTABLISHconnection, that could be due interim connectivity issues or spoofed (forged) IP source endpoints — replies it sent simply didn't reach no initiator(s).If we choose "spoofing version", I'd suggest trying to track those connections at least to the channel the server is getting them from. It might be not upstream but your LAN (be it real LAN or VMs network, etc.).
Then if it's upstream you might forward your concerns to theirs NOC (starting from support, more likely of course). If it's your side, well — clear it up.