Home / server / Questions / 453086
In Process
batfastad
Asked: 2012-11-29 08:14:20 +0800 CST

CentOS centralised logging, syslogd, rsyslog, syslog-ng, logstash sender?

  • 772

I'm trying to figure out the best way to setup a central place to store and interrogate server logs. syslog, Apache, MySQL etc.

I've found a few different options but I'm not sure what would be best. I'm looking for something that is easy to install and keep updated on many virtual machines. I can add it to a VM template going forward but I'd also like it to be easy to install to keep the VM complexity down.

The options I've found so far are:

  • syslogd
  • syslog-ng
  • rsyslog
  • syslogd/syslog-ng/rsyslog to logstash/ElasticSearch
  • logstash agent in each log "client" to send to Redis/logstash/ElasticSearch

And all sorts of permutations of the above.

What's the most resilient and light from the log "client" perspective?
I'd like to avoid the situation where log "clients" hang because they are unable to send their logs to the logging server. Also I would still like to keep local logging and the rotation/retention provided by logrotate in place.

Any ideas/suggestions or reasons for or against any of the above?
Or suggestions of a different structure entirely?

centos

4 Answers

  1. I'd say the best reliable option at this point (exluding comerial software solutions) is pure syslog-ng

    • 1

  2. I think the best is syslog-ng, easy to configure, very easy to maintain, nice and powrful tool. BTW, if you have not more than 500M logs per day, you can look into splunk, all you need is to add log-forwarder to the machine image and configure a few files, it will work as magic:) It has a lot of application for different kind of log types, nice parsing etc. All depends on what do you need.

    • 1

  3. Until very recently, we were using rsyslog with graylog for centralized logging. It worked very, very well. This uses the GELF protocol, which also has the advantage of having libraries in various programming languages.

    That said, we are switching to Loggly as it's cheaper than maintaining our own centralized logging server. Since Loggly recommends syslog-ng, we're evaluating switching to that as well, although rsyslog seems a bit better - eg. rsyslog recipes for things like log caching when networking is lost and support for GELF.

    • 0

  4. A recent rsyslog would probably be one of the fastest and lightest options. You get disk&memory buffers for free and you can output to Elasticsearch via omelasticsearch. Here's a blog post that might get you started on something like this:

    http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/

    • 0