Joerg's questions

I cannot get Squid to cache anything with the following origin response headers.

When I use curl repeatedly, I get the following response

> curl -s -D - -o /dev/null -x "http://localhost:3128" http://my-domain.com/test.jpeg
HTTP/1.1 200 OK
x-powered-by: Express
access-control-allow-origin: undefined
Vary: Accept
Content-Type: image/jpeg
Cache-Control: private, no-transform, max-age=0
Date: Thu, 11 Feb 2021 12:44:00 GMT
Server: Google Frontend
X-Cache: MISS from b6cf67dac785
X-Cache-Lookup: HIT from b6cf67dac785:3128
Transfer-Encoding: chunked
Via: 1.1 google, 1.1 b6cf67dac785 (squid/4.13)
Connection: keep-alive

It looks like it finds the image in its cache, but revalidates with the server and then gets a fresh copy from the server?

I don't want it to revalidate first - it needs to serve it straight out of the cache. I am the only client of Squid - and I want to force every single image to be cached regardless of the headers of the origin server. I know this breaks HTTP accepted protocols - but I still want to force Squid to cache absolutely everything. (it will only ever fetch images)

This is my refresh_pattern directive I am using

refresh_pattern . 525600 100% 525600 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store reload-into-ims ignore-must-revalidate ignore-private ignore-auth store-stale

I can't see which of the response headers are rendering my refresh_pattern ineffective?

I have also the following two directives set. With or without them - there is no difference.

minimum_expiry_time 1 year
max_stale 1 year

For reference, I also attach response headers that do cache well with my current refresh_pattern.

HTTP/1.1 200 OK
x-goog-generation: 1613049999992692
x-goog-metageneration: 3
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 159962
x-goog-meta-: 
x-goog-custom-time: 1970-01-01T00:00:00Z
Content-Type: image/jpeg
x-goog-hash: crc32c=6EwUgA==
x-goog-hash: md5=OGlL99uyN7kr4j0umpn5pw==
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
X-GUploader-UploadID: ABg5-UwYGdi405Mc_A1_ppi6O7NIXd3UKOpqbzcj1WvMzVN-N3U1hb-O_FdSGdBt5ZPDWU63egT-kwi4lrx8RVu4k_0
Date: Thu, 11 Feb 2021 13:27:47 GMT
Cache-Control: private, no-transform, max-age=0
Expires: Thu, 11 Feb 2021 13:27:47 GMT
Last-Modified: Thu, 11 Feb 2021 13:26:40 GMT
ETag: "38694bf7dbb237b92be23d2e9a99f9a7"
Content-Length: 0
Server: UploadServer
Age: 0
X-Cache: HIT from b6cf67dac785
X-Cache-Lookup: HIT from b6cf67dac785:3128
Via: 1.1 b6cf67dac785 (squid/4.13)
Connection: keep-alive

I am accessing images stored on storage like AWS S3 or GCS.

I am using a node library called Got to fetch an image, and I stream the response into an image processor.

I am using Got with a tunnel agent so that it uses my Squid Forward Proxy. I am going via Squid, as I want to cache the images so that I don't always need to go all the way to S3 to fetch an image.

My images, however, are not being cached. And every call to fetch the image via Squid results in the following log message:

1612789358.530   3100 172.22.0.3 TCP_TUNNEL/200 2296639 CONNECT storage.googleapis.com:80 - HIER_DIRECT/172.217.170.80 -

I have added a refresh_pattern directive to force caching.

refresh_pattern . 0 100% 525600 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store reload-into-ims ignore-must-revalidate

This should cache everything for a year.

The relevant javascript is this (in case it matters)

let stream = await got.stream(sourceUrl, {
  agent: {
    http: tunnel.httpOverHttp({
      proxy: {
        host: "localhost",
        port: "3128",
      },
    }),
  },
});

I have no experience with Squid - I am using version 4.x

Can it be, that because I am streaming responses from S3, that Squid can not cache the response once the whole image response has streamed? Or might there be a specific cache directive in my squid.conf file that is required.