I started seeing this odd sort of effect that resembles denial-of-service attack against a Linux server. The effect is that the network becomes at least partially unusable very much the same as what you see with a traditional DOS or DDOS attack.
Here's a trimmed netstat
dump from during the "attack" (assuming that's what it is):
Proto Recv-Q Send-Q Local Address Foreign Address State PID
tcp 1 0 1.2.3.1:80 50.128.251.184:1768 CLOSE_WAIT 18482/httpd
tcp 0 1 1.2.3.4:80 71.75.22.31:52323 LAST_ACK -
tcp 0 18980 1.2.3.4:80 98.180.31.210:60499 ESTABLISHED 18016/nginx: worker
tcp 0 11709 1.2.3.4:80 98.180.31.210:60498 ESTABLISHED 18016/nginx: worker
tcp 0 55743 1.2.3.4:80 71.75.22.31:52239 LAST_ACK -
tcp 0 0 1.2.3.5:80 75.190.139.103:58265 ESTABLISHED 16808/httpd
tcp 0 32814 1.2.3.4:80 71.75.22.31:52279 LAST_ACK -
tcp 0 48029 1.2.3.4:80 71.75.22.31:52284 LAST_ACK -
tcp 1 33581 1.2.3.4:80 71.75.22.31:52285 LAST_ACK -
tcp 0 23582 1.2.3.4:80 71.75.22.31:52283 LAST_ACK -
tcp 0 684 1.2.3.5:80 123.125.71.31:57865 FIN_WAIT1 -
tcp 0 37621 1.2.3.4:80 71.75.22.31:52218 LAST_ACK -
tcp 0 18980 1.2.3.4:80 174.106.209.104:39937 ESTABLISHED 18016/nginx: worker
tcp 0 0 1.2.3.1:80 95.140.125.125:60078 ESTABLISHED 18377/httpd
tcp 0 0 1.2.3.2:39509 2.2.3.1:3306 ESTABLISHED 18379/httpd
tcp 0 174 1.2.3.2:33029 2.2.3.1:3306 ESTABLISHED 18482/httpd
tcp 0 44538 1.2.3.4:80 72.230.205.217:58271 FIN_WAIT1 -
tcp 0 64812 1.2.3.2:80 184.35.67.238:49173 ESTABLISHED 1251/httpd
tcp 1 0 1.2.3.1:80 174.96.155.77:59167 CLOSE_WAIT 18379/httpd
tcp 0 1 1.2.3.4:80 174.110.137.71:61496 FIN_WAIT1 -
tcp 1 31751 1.2.3.4:80 99.25.112.12:55747 CLOSING -
tcp 0 33396 1.2.3.4:80 99.25.112.12:55764 ESTABLISHED 18016/nginx: worker
Notice primarily the high usage of Send-Q buffer space by connections that are essentially closed, or partially closed. By keeping these connections open, it appears that an attacker can burn though the allowable send queue and bring traffic largely to a halt. This doesn't appear to be a sophisticated attack, but just a few attackers can apparently bring down a server with minimal traffic.
Does anyone recognize this attack pattern and know how to counter it?
Looks like a kind of resource-exhaustion attack. Would need more data to give a more specific answer but are you able to get isolated packet captures of the attack traffic?
Not sure what you're protecting but do you need to receive many connections from the internet at large? My first attempt at countering would be that you may be able to limit how many connections are being setup - to a reasonable threshold per (host/netrange/etc).