I have quite heavy loaded web server using:
Ubuntu server
nginx
php-fpm + apc
Yesterday something wierd happened with my server. It crashed and stopped responding and after I rebooted it, web pages started loading very-very slowly, giving "request timed out" in most cases.
I've checked /var/log/syslog and saw a lot of messages like: TCP: Possible SYN flooding on port 80. Sending cookies.
Page takes about 2 minutes to load locally:
time wget -O /dev/null mysite.net
--2012-12-21 13:17:15-- http://mysite.net/
Resolving ficbook.net... 85.254.49.180
Connecting to mysite.net|85.254.49.180|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1311 (1.3K) [text/html]
Saving to: `/dev/null'
100%[========================================================================================================>] 1,311 --.-K/s in 0s
2012-12-21 13:19:18 (181 MB/s) - `/dev/null' saved [1311/1311]
real 2m2.438s
user 0m0.000s
sys 0m0.000s
I am not usre if it really is a SYN Flood attack. If so, why cookies are not helping? Here is the info from netstat:
netstat -tuna | grep :80 | grep SYN_RECV
tcp 0 0 85.254.49.180:80 92.37.173.66:3214 SYN_RECV
tcp 0 0 85.254.49.180:80 81.26.91.4:49471 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4940 SYN_RECV
tcp 0 0 85.254.49.180:80 213.87.140.242:23259 SYN_RECV
tcp 0 0 85.254.49.180:80 94.139.229.219:49827 SYN_RECV
tcp 0 0 85.254.49.180:80 95.67.233.125:51267 SYN_RECV
tcp 0 0 85.254.49.180:80 83.149.2.69:7051 SYN_RECV
tcp 0 0 85.254.49.180:80 95.67.239.40:54497 SYN_RECV
tcp 0 0 85.254.49.180:80 195.91.229.193:58981 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4925 SYN_RECV
tcp 0 0 85.254.49.180:80 88.154.3.228:59086 SYN_RECV
tcp 0 0 85.254.49.180:80 92.113.26.124:3887 SYN_RECV
tcp 0 0 85.254.49.180:80 77.34.83.254:26963 SYN_RECV
tcp 0 0 85.254.49.180:80 195.208.64.130:3542 SYN_RECV
tcp 0 0 85.254.49.180:80 81.26.91.4:49480 SYN_RECV
tcp 0 0 85.254.49.180:80 87.253.29.234:53130 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4939 SYN_RECV
tcp 0 0 85.254.49.180:80 88.154.3.228:41696 SYN_RECV
tcp 0 0 85.254.49.180:80 178.45.39.169:41758 SYN_RECV
tcp 0 0 85.254.49.180:80 217.118.66.37:51534 SYN_RECV
tcp 0 0 85.254.49.180:80 83.149.9.197:8249 SYN_RECV
tcp 0 0 85.254.49.180:80 37.29.88.202:3531 SYN_RECV
tcp 0 0 85.254.49.180:80 178.34.206.52:3409 SYN_RECV
tcp 0 0 85.254.49.180:80 193.188.254.93:50317 SYN_RECV
tcp 0 0 85.254.49.180:80 217.66.152.162:8883 SYN_RECV
tcp 0 0 85.254.49.180:80 109.198.235.10:56382 SYN_RECV
tcp 0 0 85.254.49.180:80 95.53.159.39:2256 SYN_RECV
tcp 0 0 85.254.49.180:80 188.232.13.175:49819 SYN_RECV
tcp 0 0 85.254.49.180:80 88.203.2.27:64080 SYN_RECV
tcp 0 0 85.254.49.180:80 217.118.64.52:12382 SYN_RECV
tcp 0 0 85.254.49.180:80 92.124.76.189:3416 SYN_RECV
tcp 0 0 85.254.49.180:80 37.29.88.202:30532 SYN_RECV
tcp 0 0 85.254.49.180:80 87.253.29.234:53131 SYN_RECV
tcp 0 0 85.254.49.180:80 213.87.123.1:44943 SYN_RECV
tcp 0 0 85.254.49.180:80 176.51.255.3:1642 SYN_RECV
tcp 0 0 85.254.49.180:80 85.26.165.112:56906 SYN_RECV
tcp 0 0 85.254.49.180:80 88.203.2.27:64081 SYN_RECV
tcp 0 0 85.254.49.180:80 217.118.66.37:51533 SYN_RECV
tcp 0 0 85.254.49.180:80 176.51.211.131:1699 SYN_RECV
tcp 0 0 85.254.49.180:80 37.29.88.202:22233 SYN_RECV
tcp 0 0 85.254.49.180:80 211.167.112.18:58353 SYN_RECV
tcp 0 0 85.254.49.180:80 217.118.66.32:38640 SYN_RECV
tcp 0 0 85.254.49.180:80 217.144.185.150:64421 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4928 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4927 SYN_RECV
tcp 0 0 85.254.49.180:80 94.153.254.218:1084 SYN_RECV
tcp 0 0 85.254.49.180:80 37.29.88.202:30384 SYN_RECV
tcp 0 0 85.254.49.180:80 46.201.3.189:51032 SYN_RECV
tcp 0 0 85.254.49.180:80 109.187.107.41:50565 SYN_RECV
tcp 0 0 85.254.49.180:80 91.146.60.86:49266 SYN_RECV
tcp 0 0 85.254.49.180:80 87.253.29.234:53134 SYN_RECV
tcp 0 0 85.254.49.180:80 80.83.238.25:2515 SYN_RECV
tcp 0 0 85.254.49.180:80 176.102.16.8:54291 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4918 SYN_RECV
tcp 0 0 85.254.49.180:80 95.153.164.165:26752 SYN_RECV
tcp 0 0 85.254.49.180:80 80.83.239.76:46519 SYN_RECV
tcp 0 0 85.254.49.180:80 94.139.229.219:49826 SYN_RECV
tcp 0 0 85.254.49.180:80 188.239.193.48:49418 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4919 SYN_RECV
tcp 0 0 85.254.49.180:80 217.118.66.32:38639 SYN_RECV
tcp 0 0 85.254.49.180:80 95.67.233.125:51266 SYN_RECV
tcp 0 0 85.254.49.180:80 85.26.235.172:59092 SYN_RECV
tcp 0 0 85.254.49.180:80 213.87.136.21:44804 SYN_RECV
tcp 0 0 85.254.49.180:80 95.109.193.247:1206 SYN_RECV
tcp 0 0 85.254.49.180:80 217.112.11.130:2714 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4941 SYN_RECV
tcp 0 0 85.254.49.180:80 88.154.3.228:52640 SYN_RECV
tcp 0 0 85.254.49.180:80 37.79.93.27:64801 SYN_RECV
tcp 0 0 85.254.49.180:80 91.203.96.76:45132 SYN_RECV
tcp 0 0 85.254.49.180:80 80.83.238.25:2513 SYN_RECV
tcp 0 0 85.254.49.180:80 85.26.235.172:60092 SYN_RECV
tcp 0 0 85.254.49.180:80 188.239.193.48:49416 SYN_RECV
tcp 0 0 85.254.49.180:80 178.130.42.68:60373 SYN_RECV
tcp 0 0 85.254.49.180:80 80.239.243.181:58110 SYN_RECV
tcp 0 0 85.254.49.180:80 87.253.29.234:53128 SYN_RECV
tcp 0 0 85.254.49.180:80 83.149.9.197:18870 SYN_RECV
tcp 0 0 85.254.49.180:80 88.154.3.228:53380 SYN_RECV
tcp 0 0 85.254.49.180:80 88.135.63.40:58845 SYN_RECV
tcp 0 0 85.254.49.180:80 80.239.243.110:52234 SYN_RECV
tcp 0 0 85.254.49.180:80 46.201.3.189:51028 SYN_RECV
tcp 0 0 85.254.49.180:80 88.154.3.228:53457 SYN_RECV
tcp 0 0 85.254.49.180:80 85.235.176.138:12101 SYN_RECV
tcp 0 0 85.254.49.180:80 109.187.107.41:50567 SYN_RECV
tcp 0 0 85.254.49.180:80 83.149.48.29:4172 SYN_RECV
tcp 0 0 85.254.49.180:80 188.232.13.175:49820 SYN_RECV
tcp 0 0 85.254.49.180:80 37.29.88.202:6651 SYN_RECV
tcp 0 0 85.254.49.180:80 91.198.143.6:45591 SYN_RECV
tcp 0 0 85.254.49.180:80 85.235.176.138:50667 SYN_RECV
tcp 0 0 85.254.49.180:80 176.209.98.72:53653 SYN_RECV
tcp 0 0 85.254.49.180:80 80.83.239.71:49701 SYN_RECV
tcp 0 0 85.254.49.180:80 188.232.13.175:49817 SYN_RECV
tcp 0 0 85.254.49.180:80 188.239.193.48:49417 SYN_RECV
tcp 0 0 85.254.49.180:80 88.154.3.228:54175 SYN_RECV
tcp 0 0 85.254.49.180:80 61.147.79.111:51039 SYN_RECV
tcp 0 0 85.254.49.180:80 88.154.3.228:58854 SYN_RECV
tcp 0 0 85.254.49.180:80 87.253.29.234:53135 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4938 SYN_RECV
tcp 0 0 85.254.49.180:80 62.122.51.139:4942 SYN_RECV
tcp 0 0 85.254.49.180:80 176.209.98.72:53662 SYN_RECV
tcp 0 0 85.254.49.180:80 2.74.51.158:1092 SYN_RECV
tcp 0 0 85.254.49.180:80 213.87.140.242:48178 SYN_RECV
tcp 0 0 85.254.49.180:80 213.87.129.42:29549 SYN_RECV
tcp 0 0 85.254.49.180:80 37.29.88.202:28428 SYN_RECV
tcp 0 0 85.254.49.180:80 85.26.235.172:50983 SYN_RECV
tcp 0 0 85.254.49.180:80 217.118.64.52:12381 SYN_RECV
tcp 0 0 85.254.49.180:80 85.26.235.172:55459 SYN_RECV
tcp 0 0 85.254.49.180:80 84.244.12.209:64975 SYN_RECV
tcp 0 0 85.254.49.180:80 83.149.2.121:10768 SYN_RECV
tcp 0 0 85.254.49.180:80 84.240.248.206:3494 SYN_RECV
tcp 0 0 85.254.49.180:80 195.91.229.193:52428 SYN_RECV
tcp 0 0 85.254.49.180:80 95.109.193.247:1202 SYN_RECV
tcp 0 0 85.254.49.180:80 79.105.204.56:56822 SYN_RECV
tcp 0 0 85.254.49.180:80 85.15.184.141:56335 SYN_RECV
tcp 0 0 85.254.49.180:80 164.177.225.31:50584 SYN_RECV
tcp 0 0 85.254.49.180:80 80.83.238.25:2511 SYN_RECV
tcp 0 0 85.254.49.180:80 84.240.248.206:3493 SYN_RECV
tcp 0 0 85.254.49.180:80 80.83.239.76:26950 SYN_RECV
tcp 0 0 85.254.49.180:80 84.240.248.206:3495 SYN_RECV
tcp 0 0 85.254.49.180:80 217.144.185.150:58141 SYN_RECV
tcp 0 0 85.254.49.180:80 178.215.97.15:13346 SYN_RECV
I've tried disabling syn cookies, but it had no effect. Seems, that server is limiting connection count, if you look at metrics "hits per minute" it looks like this:
Esterday, before crash, everyting worked fine. I would appriciate any info or advice on what could be the problem or how to diagnose it.
UPDATE
I am pretty sure it is not an attack. When I restart nginx - everything works for few hours and then, again syslog is full of:
Possible SYN flooding on port 80
Possible SYN flooding on port 9000
And nginx error log first get a lot of 104 errors:
2013/01/08 20:28:24 [error] 959#0: *2387458 recv() failed (104: Connection reset by peer) while reading response header from upstream
And then 110:
2013/01/08 21:27:19 [error] 30349#0: *760749 upstream timed out (110: Connection timed out) while connecting to upstream
This happens in evenings, when load reaches certain amount of hist (about 800 per second) and something goes wrong.
Turning syn cookies off and adjusting backlog give no effect.
Internet is full of similar claims, but no real answer can be found. Please help!
It sounds like there's something wrong with your upstream server that makes nginx appear very slow.
When nginx is appearing slow, do requests take a long time only if they're proxied over to your php-fpm + apc? Have you tried defining a non-proxied
location, and seeing whether that exhibits any problems?Is your php-fpm + apc running out of memory, out of connections / file-descriptors or out of worker threads / processes? Are you using OpenVZ? Or any other potentially broken-by-design kernel-level virtualisation? If not, do you have any other process or memory limits that you might be hitting? You can check for limits by doing an
suinto the user which runs your php-fpm / apc, and runninglimitintcsh.You should probably post your entire configs for nginx and php-fpm + apc, otherwise, it's a big guessing game. I'm not a php guy, but my educated guess is that you have some kind of connection or worker thread limit on your php-fpm+apc side, which your nginx proxy is exceeding.
Also, I see you have a nice graph of how things went downhill one day all of a sudden; did you make any recent changes or upgrades within a couple of days before that happened?
Is it possible to rate limit the incoming connections using IPtables or nginx ?
I strongly recommend you to setup some default Iptables chains. Try to drop SYN floodings and log all dropped packets. You even don't know what the heck is happening on your network layer.
Finally, it's simple for somebody who has experience with network analysis to detect your server crash and to stop the attack immediately.