Home / server / Questions / 734899
In Process
Denis Pshenov
Asked: 2015-11-10 00:53:03 +0800 CST

Is this an nginx proxy_cache bug?

  • 772

I have a very simple proxy config:

http {
    proxy_cache_path /var/www/cache levels=1:2 keys_zone=s3-images-cache:50m inactive=1M max_size=1000m;
    proxy_temp_path /var/www/cache/tmp;

    server {
        listen 80;
        server_name images.example.com;

        location / {
            proxy_cache s3-images-cache;
            proxy_cache_key $scheme$proxy_host$uri$is_args$args;
            proxy_cache_bypass $http_purge_cache;
            proxy_cache_valid any 1y;
            proxy_pass http://images-example.s3.amazonaws.com;
            add_header X-Cache $upstream_cache_status;

            proxy_intercept_errors on;
            error_page 404 = @no_image;
        }

        location @no_image {
            return 403;
        }
    }
}

Now follow me here:

  1. Let's request /image.jpg.
  2. Request is sent to proxy for /image.jpg (does not exists yet).
  3. Backend responds with 404.
  4. "proxy_intercept_errors on" kicks in and "error_page 404 = @no_image" is called.
  5. Nginx returns 403.
  6. Do another request for same image and see that "X-Cache: HIT" is set. We are clearly hitting proxy cache.

But, if we check /var/www/cache/ folder at this time we will see that there is no cache item created for this request. So does it mean Nginx keeps the cache for it in memory and forgot to write to file?

  1. Let's upload /image.jpg to backend.
  2. Now do "PURGE-CACHE: 1" request to that image. We see that now we get the image instead of 403 with "X-Cache: BYPASS" header present. Good.

If we check /var/www/cache/ we will see that the cache file is now finally created for this request. Looking inside the cached file we see that it is our image.

  1. Now here is the problem: lets request /image.jpg again with a normal GET request. We should get the newly uploaded image right?
  2. Nginx returns 403 with "X-Cache: HIT". Why? It seems like it's hitting the cache but returning something else not what is in /var/www/cache folder?? How?

My only explanation to this is it seems that Nginx is caching the response in memory and doesn't write to file when we are hitting error with our custom error_page in the proxied responses. Furthermore when using proxy_cache_bypass it does not overwrite in-memory cache, so that subsequent requests to the same item will be using old cache which is stored in memory and not the new one created in the cache folder.

Could someone please let me know if I am doing something wrong or this really is a bug. Spent last 3 days fighting this.

UPDATE: Backend returns normal set of headers you would would expect from S3 in both 200 and 404 responses:

404

Connection:close
Content-Type:application/xml
Date:Fri, 20 Nov 2015 07:41:39 GMT
Server:AmazonS3
Transfer-Encoding:chunked
x-amz-id-2:bH8L/1dOVGShsGJdZZ/zS/X6UkHS+KMAxDxnPvOkIalpPphFJXr9zZ1RiV6L2a13NXoZ3QdCOeE=
x-amz-request-id:D66FDBFAA9643252

200

Accept-Ranges:bytes
Connection:keep-alive
Content-Length:10533
Content-Type:image/jpeg
Date:Fri, 20 Nov 2015 07:47:12 GMT
ETag:"061b4dae0b2bbdf4a4fa212951f4ba79"
Last-Modified:Wed, 11 Nov 2015 14:29:09 GMT
Server:AmazonS3
x-amz-id-2:qsSmH/gkvql2jnj67p0vguZBXQJHfS+Yk70llBaDvbgH0xSCbvj9G9JlKn5WhWTdty0+JzApN7k=
x-amz-request-id:8CF04EA869190E63
nginx

0 Answers