Home / server / Questions / 290776
In Process
WerkkreW
Asked: 2011-07-16 08:22:01 +0800 CST

Mod_pagespeed, Varnish and Apache cache issues after new code pushes

  • 772

I have a rather strange issue. In my environment we are running a load balanced cluster of 8 apache servers with a master-master MySQL backend. In front of apache we have Varnish in the cache layer.

We have been running Apache mod_pagespeed for several weeks now and for the most part it has been working great. The issue arises when we do fresh code updates from Git, and and/all of the JS/CSS assets change.

Basically the problem appears to be two fold. One, after the code push we generally take the opportunity to flush varnish, restart apache, and restart varnish. In doing this all of the mod_pagespeed combinied/minified files are cleared out ensuring that all of the new JS/CSS assets are fresh. The problem is, upon doing this the file names that mod_pagespeed creates change, but the old files (appear) to be still cached for many people client side leading to very unexpected results. However, if we do not restart apache, the changes to the files may or may not appear client side due to the cached minified assets.

The simple solution is to disable mod_pagespeed, however I would rather not do that as it has made a fairly large impact in performance.

I feel as if there must be a better way to deal with the inconsistencies in cache between the client and server to prevent having people to go to great lengths or perform a large number of page refreshes to see a working page.

I can provide configuration snippets if anyone needs them.

If you would like to inspect the site, source, headers, or anything try the following addresses:

http://wellplayed.org

http://wellplayed.org/tv

Thanks in advance!

apache-2.2 varnish git mod-pagespeed

2 Answers

  1. Just to make sure I understand the problem it seems like you are modifying files that were served up to mod_pagespeed from varnish with expires headers that are keeping it cached beyond when it has been changed on the backend.

    You have a couple options here if that is the case. You can configure your backend(varnish) to send shorter expires headers. Or you can configure your non-cached assets to link to the cached versions with patch numbers that you can increment when you make changes. So they become:

    http://site.com/asset.gif?p=930024982afb191

    It also looks like there are other knobs you can tweek in mod_pagespeed to change expiry times for certain files:

    http://code.google.com/speed/page-speed/docs/filter-cache-extend.html

    Also in relation to the client side, here is more info on how they act that will help configuring both varnish and mod_pagespeed:

    http://code.google.com/speed/page-speed/docs/filter-cache-extend.html

    • 1

  2. You wrote about flushing Varnish. Actually there are two different ways to do that:

    1. remove a specific document (HTML page, image, CSS file or whatever) based on a specific URL and it's variants (usually different language versions or compressed/uncompressed versions) - this is mostly called purge
    2. void a group of documents (HTML page, image, CSS file or whatever) based on a regular expression - this is called ban

    In your case a ban off all HTML, CSS and JS files could be a good idea.

    For more information please follow this link or ask: https://www.varnish-cache.org/docs/3.0/tutorial/purging.html

    • 1