Home / server / Questions / 435291
In Process
Justin Hourigan
Asked: 2012-10-06 08:15:42 +0800 CST

Varnish Cache logging to MongoDB

  • 772

I want to stream Varnish's log into MongoDB for realtime reporting.

Can this be configured directly within Varnish or can the official Varnish-Agent (https://github.com/varnish/varnish-agent) be used or some other agent or script?

logging

3 Answers

  1. I've also thought about logging statistics into mongoDB since it has a "fire and forget" as default write mode and it would not block requests but never got further with it.

    I see 3 possible solutions:

    1. There's no mongoDB vmod available yet (https://www.varnish-cache.org/vmods) and this may be a kickstart for one.

    2. Use the mongoDB C lib (http://www.mongodb.org/display/DOCS/C+Language+Center) and write a inline C inside your VCL using the runtime functions (VRT_GetHdr and etc).

    3. If you're gonna save few stats as page visits and error codes, you can parse the log from varnishncsa with a python script and store info in mongoDB for latter usage (map and reduce). But this is restricted to a Apache's common log format (there's not much information in there).

    • 1

  2. I decided to use a PHP script with ran a command the varnishstat -1 command via SSH. I based it on this project https://github.com/redsnapper8t8/pyvarnish

    • 0

  3. You don't really want it done in Varnish. What you want is to do have something like varnishncsa talk directly to mongodb. That way if Mongo slows down page delivery isn't slowed down. We're currently playing with idea of building a varnishlogjson program to output (parts of the) varnishlog in JSON format, for insertion into hadoop, but maybe Mongo would be able to eat JSON as well.

    • 0