Home / server / Questions / 297900
Accepted
Alienfluid
Asked: 2011-08-05 19:51:41 +0800 CST

Real-time log parsing and reporting

  • 772

We have a small project we are working on part-time that runs on Nginx/MongoDB on Ubuntu 10.04 LTS Server. We'd like to be able to see reports on things like server load, requests/sec, response time, DB load, DB response time, etc. Is there an open source or free (as in beer) tool that can parse such logs and provide a real-time report?

I looked into Splunk briefly, but I wanted to see if there are any others that are highly recommended.

http logging nginx mongodb parse-server

4 Answers

  1. Best Answer

    Most of the metrics you would like to see are accessible through the respective applications statistics, which is much more effective than log analysis. And there are several free projects aimed to collect such metrics, build graphs, check margins, send warnings, etc. All of them are designed to be easily extensible, so that a lot of plugins for collecting metrics exist and you can write your own. The names of the main projects I remember of are Munin, Cacti and Graphite, but in fact there are more.

    Munin is one of the oldest and has a lot of plugins even in the standard package. From your list:

    • server load Munin has plugins to show load average (LA), iostat, vmstat
    • Nginx Munin has "nginx_request" (req/sec) and "nginx_status" (number of connections in a specific status, i.e. active, reading, writing)
    • response time As far as I know, there is no such plugin. See this question for more information
    • MongoDB MongoDB team provides several Munin plugins at GitHub
    • 3

  2. You can do it with a shell script and push the result to Ganglia gmetric or Nagios + PNP4Nagios.

    • 0

  3. AFAIK there's nothing available for measuring requests/sec and response times. An answer to what tools you might use to capture and report on these metrics alone would fill a large book!

    For Mongo, the choice is bit simpler. Assuming you don't want to roll-your-own from scratch, then have a look at Server Desnity's (hosting company) monitoring toolkit

    • 0

  4. You might want to take a look at our NGINX sFlow module along with sFlowTrend.

    NGINX sFlow module

    sFlowTrend adds web server monitoring

    • 0