Home / server / Questions / 482875
Accepted
Julien
Asked: 2013-02-27 20:48:49 +0800 CST

Why does Nginx remove Content-Length header for chunked content?

  • 772

I use nginx 1.2.3 to proxy to a script:

proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8880;
proxy_buffering off;
proxy_read_timeout 300s;
gzip off;

The scripts sends both Transfer-encoding: chunked and Content-Length: 251:

HTTP/1.0 307 Temporary Redirect
Content-length: 251
Pragma: no-cache
Location: /...
Cache-control: no-cache
Transfer-encoding: chunked

I need both, but nginx automatically removes the Content-Length:

HTTP/1.1 302 Found
Server: nginx/1.2.3
Content-Type: application/json; charset=utf-8
Content-Length: 58
Connection: keep-alive
Location: /...

As a result, the clients do not wait for the chunks to be sent. This used to work with an earlier version of nginx.

nginx

5 Answers

  1. Best Answer

    Unfortunately, I can't comment on cnst's post - so I'm going to answer here.

    The nginx_http_proxy module by default talks with the upstream in HTTP/1.0. This can be changed with the directive proxy_http_version 1.1.

    This might also be the cause for your script to return a HTTP/1.0 answer, although chunked coding and status code 307 don't exist in this version.

    You shouldn't use chunked coding with a redirect either, as this doesn't really make sense.

    Additionally, it seems like nginx doesn't pass chunks from the upstream to the client one by one, but it buffers the upstream's response. The Content-Length header field is ignored because it is against the definition. I had to look at the source code of the module because all this appears to be undocumented.

    You may want to try out the nginx_tcp_proxy_module to proxy the chunked content as raw TCP data: Module at Github

    UPDATE (10.04.14)
    The nginx_http_proxy module has support for X-Accel-* headers, of which one (X-Accel-Buffering: yes|no) controls whether the response should be buffered or not.

    Adding this header (X-Accel-Buffering: no) to the backend's response will cause nginx to directly pass chunks to the client.

    This header allows to control buffering on an per-request basis.

    The module also has a configuration directive proxy_buffering to enable or disable response buffering (not buffering means sending chunks will work).

    Proxy buffering (both header and directive based) is documented here.

    • 12

  2. As Lukas alludet to, HTTP 1.1 prohibits Content-Length if there's a Transfer-Encoding set.

    Quoting http://www.ietf.org/rfc/rfc2616.txt:

       3.If a Content-Length header field (section 14.13) is present, its
     decimal value in OCTETs represents both the entity-length and the
     transfer-length. The Content-Length header field MUST NOT be sent
     if these two lengths are different (i.e., if a Transfer-Encoding
     header field is present). If a message is received with both a
     Transfer-Encoding header field and a Content-Length header field,
     the latter MUST be ignored.
    • 10

  3. You have not specifically elaborated why your script needs chunked encoding in the first place, especially with a redirect response.

    I see a multitude of problems here.

    • Transfer-Encoding: chunked is an HTTP/1.1 feature (and your script seems to be replying with an HTTP/1.0 header)

    • there is no 307 in HTTP/1.0

    • the whole purpose of chunked is that you don't know what your Content-Length would have been, so, chunked is used in place of providing the length within Content-Length, where instead lengths are provided within the body of the response, intermixed with the actual content; it would be pointless for a script to be generating both headers upfront

    I'm not personally familiar with chunked, but as per the basic info at http://en.wikipedia.org/wiki/Chunked_transfer_encoding and also https://www.rfc-editor.org/rfc/rfc2616#section-3.6.1, I would guesstimate that your script's whole handling of the chunked encoding may be completely wrong.

    If the above still doesn't cover it, and in all actuality otherwise, it is also unclear why a reply with a 307 or 302 http status code should be provided with a "weird" encoding. There was recently a similar discussion in the nginx mailing list about 410 Gone and other error pages always excluded from gzip compression, and I think the sentiment would equally apply here. (http://mailman.nginx.org/pipermail/nginx/2013-March/037890.html)

    • 3

  4. I had the same issue streaming mp4 file through html5 video tag.

    Safari and Firefox behaved normally whereas Chrome was triggering ERR_CONTENT_LENGTH_MISMATCH at some point (but it allowed me to wach several minutes of the video before failing).

    The problem didn't reproduce after I turned off cache control for mp4 files.

    • 1

  5. Sharing this answer I posted to SO in case it's helpful: https://stackoverflow.com/questions/50499637/mp4-video-safari-cloudflare-nginx-rails-no-play/59348509#59348509

    I had a similar issue with mp4 playback due to chunks not being served, and confirmed the issue per Apple's guide, listed below. I verified I was downloading the whole file, and after the fix below, only the first chunk.

    curl --range 0-99 http://example.com/test.mov -o /dev/null

    I resolved my Safari .mp4 playback by changing my gzip compression settings in my nginx.conf, to remove gzip compression of .mp4 files.

    Here's the block in nginx for reference. (Note: depending on how your app is configured, you may need to change the location line to location ~ \.mp4$ { 

    location ~ ^/(assets|system|videos)/  {
   expires max;
   add_header Cache-Control public;
   add_header ETag "";
   gzip on;
   gzip_http_version 1.1;
   gzip_vary on;
   gzip_comp_level 6;
   gzip_proxied any;

   # Reference configuration
   #gzip_types text/plain text/html text/css application/json application/javascript application/x-javascript text/javascript video/mp4 application/mp4 image/jpeg image/png image/svg+xml application/x-font-ttf application/x-font-truetype application/font-woff application/font-woff2 application/vnd.ms-fontobject;

   # Kelton trying to fix cloudflare by removing the mp4 settings
   gzip_types text/plain text/html text/css application/json application/javascript application/x-javascript text/javascript image/jpeg image/png image/svg+xml application/application/x-font-ttf application/x-font-truetype application/font-woff application/font-woff2 application/vnd.ms-fontobject;
}

    Link to Apple documentation reference: https://developer.apple.com/library/archive/documentation/AppleApplications/Reference/SafariWebContent/CreatingVideoforSafarioniPhone/CreatingVideoforSafarioniPhone.html#//apple_ref/doc/uid/TP40006514-SW6

    • 0