I'm having this problem where when I start up nginx it starts sending/receiving traffic at a rate of 1 GET
request every second from nginx on port 80
to /ping
, but I can't track down what is causing it. Any ideas on how to track down what process is causing this traffic?
Log samples: /tmp/nginx.error.log
2011/03/31 21:30:33 [error] 6587#0: *7213 connect() to unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 10.252.27.111, server: myhost.com, request: "GET /ping HTTP/1.1", upstream: "http://unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock:/ping", host: "newcaprica11"
2011/03/31 21:30:33 [error] 6587#0: *7215 connect() to unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 10.204.203.236, server: myhost.com, request: "GET /ping HTTP/1.1", upstream: "http://unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock:/ping", host: "newcaprica11"
2011/03/31 21:30:34 [error] 6587#0: *7217 connect() to unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 10.252.27.111, server: myhost.com, request: "GET /ping HTTP/1.1", upstream: "http://unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock:/ping", host: "newcaprica11"
2011/03/31 21:30:34 [error] 6587#0: *7219 connect() to unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 10.204.203.236, server: myhost.com, request: "GET /ping HTTP/1.1", upstream: "http://unix:/home/deploy/apps/dashboard/current/tmp/sockets/unicorn.sock:/ping", host: "newcaprica11"
Nginx access Log: nginx.vhost.access.log
10.252.27.111 - - [31/Mar/2011:21:33:23 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
10.204.203.236 - - [31/Mar/2011:21:33:23 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
10.252.27.111 - - [31/Mar/2011:21:33:24 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
10.204.203.236 - - [31/Mar/2011:21:33:24 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
10.252.27.111 - - [31/Mar/2011:21:33:25 +0000] "GET /ping HTTP/1.1" 502 728 "-" "-"
I replaced our real hostname with myhost.com, but the strange thing is that in the error.log there is that host host: "newcaprica11"
More details:
nmap -P0 10.204.203.236
Starting Nmap 5.21 ( http://nmap.org ) at 2011-03-31 21:38 UTC
Nmap scan report for ip-10-204-203-236.ec2.internal (10.204.203.236)
Host is up (0.00042s latency).
Not shown: 994 filtered ports
PORT STATE SERVICE
80/tcp open http
6667/tcp closed irc
8001/tcp closed unknown
8080/tcp closed http-proxy
9000/tcp closed cslistener
9090/tcp closed zeus-admin
So they are running some sort of webservice on 80
Additionally a telnet reveals:
telnet 10.204.203.236 80
Trying 10.204.203.236...
Connected to 10.204.203.236.
Escape character is '^]'.
GET / HTTP/1.0
HTTP/1.1 302 Found
Server: Apache/2.2.9 (Debian) Phusion_Passenger/2.2.15
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.15
X-Runtime: 310
Cache-Control: no-cache
Location: http://newcaprica9/neighborhoods
Status: 302
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
Content-Length: 98
Date: Thu, 31 Mar 2011 21:39:16 GMT
X-Varnish: 2154834965
Age: 0
Via: 1.1 varnish
Connection: close
<html><body>You are being <a href="http://newcaprica9/neighborhoods">redirected</a>.</body></html>Connection closed by foreign host.
So clearly its some Rails or Rack app running something shitty.
From the log files, it looks a lot like the servers
10.252.27.111
and10.204.203.236
are requesting the/ping
resource. Nginx is complaining because/ping
is configured within Nginx to go and connect to a local socketunicorn.sock
and pull data from there to fulfill the request. It's getting aconnection refused
from the socket so is throwing the 502 error.In this context, I believe upstream is the backend that nginx is connecting to in order to get the response for the request.
I'm not sure if you've anonymised those IP addresses, or if they really are RFC1918 addresses. Either way, they're the ones doing the requesting.
I think.