Home / server / Questions / 406887
Accepted
rush
Asked: 2012-07-12 10:52:36 +0800 CST

nginx closes connection on some pictures

  • 772

There is problem with nginx. It closes connection before client finishes download. It looks like:

 $ wget -O /dev/null http://www.site.com/images/theme/front/clean.jpg
--2012-07-11 21:37:03--  http://www.site.com/images/theme/front/clean.jpg
Resolving www.site.com (www.site.com)... 123.234.123.234
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90707 (89K) [image/jpeg]
Saving to: `/dev/null'

26% [===============>                    ] 24,291      --.-K/s   in 8.7s    

2012-07-11 21:37:12 (2.74 KB/s) - Connection closed at byte 24291. Retrying.

--2012-07-11 21:37:13--  (try: 2)  http://www.site.com/images/theme/front/clean.jpg
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 90707 (89K), 66416 (65K) remaining [image/jpeg]
Saving to: `/dev/null'

53% [+++++++++++++++============>        ] 48,555      --.-K/s   in 8.7s    

2012-07-11 21:37:23 (2.74 KB/s) - Connection closed at byte 48555. Retrying.

--2012-07-11 21:37:25--  (try: 3)  http://www.site.com/images/theme/front/clean.jpg
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 90707 (89K), 42152 (41K) remaining [image/jpeg]
Saving to: `/dev/null'

100%[+++++++++++++++++++++++++++========>] 90,707      --.-K/s   in 0.1s    

2012-07-11 21:37:25 (311 KB/s) - `/dev/null' saved [90707/90707]

at the same time with smaller images all is ok: 

 $ wget -O /dev/null http://www.site.com/images/theme/front/grease.jpg
--2012-07-11 21:41:28--  http://www.site.com/images/theme/front/grease.jpg
Resolving www.site.com (www.site.com)... 123.234.123.234
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21404 (21K) [image/jpeg]
Saving to: `/dev/null'

100%[====================================>] 21,404      --.-K/s   in 0.07s   

2012-07-11 21:41:29 (316 KB/s) - `/dev/null' saved [21404/21404]

This is the reason why this picture can't bee drawn fully size in browser. I can see only head of it.

Nginx is configured as front-end and apache as back-end. Direct link to apache works well, so there is problem in nginx. Am I right?

nginx config:

user satellite;
worker_processes  1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    access_log  /var/log/nginx/access.log;

    sendfile        on;
    keepalive_timeout  0;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    client_max_body_size 100m;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
    server {
            listen 123.234.123.234:80;
            server_name site.com www.site.com;
            location ~* ^/(admin/|dump/|webmail/|myadmin/|webim/) {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_redirect http://site.com:8080/ /;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
            }
            location / {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_redirect http://site.com:8080/ /;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
                    proxy_cache ino;
                    proxy_cache_valid 12h;
                    proxy_hide_header "Set-Cookie";
                    proxy_ignore_headers "Cache-Control" "Expires";
            }
            location ~* ^.+\.(jpg|swf|flv|ico|txt|jpeg|gif|png|svg|js|css|mp3|ogg|mpe?g|avi|zip|gz|bz2?|rar)$ {
                    access_log /home/satellite/logs/site.com.nginx.access.log;
                    error_page 404 = @fallback;
                    if ( $host ~* ^((.*).site.com)$ ) {
                            set $proot /home/satellite/www/$1;
                            break;
                    }
                    if ( $host = "www.site.com" ) {
                            break;
                    }
                    if ( $host = "site.com" ) {
                            break;
                    }

                    root /home/satellite/www/site.com;
            }
            location @fallback {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
            }
    }

where I should dig to fix this issue?

web-server nginx

5 Answers

  1. Best Answer

    The main thing I forgot is to check /var/log/nginx/error.log.

    2012/07/12 08:46:44 [crit] 24074#0: *3 open() 
"/var/lib/nginx/proxy/1/00/0000000001" failed (13: Permission denied) 
while reading upstream, client: 109.173.96.30, server: site.com, request: 
"GET /images/theme/front/clean.jpg HTTP/1.1", upstream: 
"http://123.234.123.234:8080/images/theme/front/clean.jpg", 
host: "www.site.com", referrer: "http://www.google.com"

    So I fixed /var/lib/nginx/proxy/* directories permissions (sudo chown -R www-data:www-data /var/lib/nginx/proxy/*) and now everything works great. Thank everybody for help.

    • 10

  2. A possible reason for the closing of the connection is a slow connection and a short keepalive_timeout. The default value for the keepalive_timeout is 75s. If it is too short and the connection is slow, then it could be closed too early. 

    http {
    ..
    keepalive_timeout 75;
}

    One reason why your image download could be slow is your application. If you are using a Ruby-on-Rails application with an Asset Pipeline in combination with Nginx, then the image download could be slow because you are using the wrong image urls (without fingerprint generated by the asset pipeline). The Rails helpers asset_path and image_tag generate the right urls form files with fingerprint which can be downloaded quickly.

    • 1

  3. Another very important thing to check is: Make sure you have disk space left!

    For me it was like following:

    [user@server]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        30G   29G     0 100% /

    For me, nginx was unable to write to the disk eventually causing the same issue! Hope it helps someone!

    • 1

  4. Your download rate is incredibly low. (2.74 KB/s!). Do you get the same result when running wget on the same machine where Nginx is located? It could be that Nginx is reacting reasonably to a request over a very slow link.

    Otherwise, I recommend exploring the various time directives in the Nginx docs. Search for every mention of "timeout" on the page and see if you find that's a good match. For example, you can see you are timing out in what appears to be 10 seconds intervals, so any timeout that's about 10 seconds should receive extra scrutiny.

    For example, the lingering_timeout directive defaults to 10 seconds, so you might check that.

    You should also be looking into why the connection with your client is apparently so slow.

    Another idea: Try an alternate client, like curl, and see you get the same result as you do with wget. If curl works fine, you should suspect there's something odd about wget is making the request.

    • 0

  5. Check the lingering_time value in nginx.conf. This will be by default set to 30sec. So what this will do is that, nginx will wait maximum of 30 sec for the client to send data.

    If you are doing an upload or POST of a file which might take more than 30secs to complete, then the nginx server will reset the connection to client by sending a RST packet to client if the time to upload exceeds 30 secs.

    For the nginx to wait for more time to listen to client data, then set this value to a higher value.

    For detailed info regarding lingering_time, see it here lingering_time

    • 0