I have this problem or rather an itch that I could really use help with. I use autoscale and several instances on Amazon. Problem is I have all these logs PHP, Apache, system but also my web app generating traffic logs etc etc. I want to consolidate all these logs. I know syslogd etc exist, but I was wondering whats the most effective , low administrative solution devops are using currently since my experience on Amazon AWS isn't that extensive. Plus I need the solution to be low resource usage if possible. Hitting database to save all that data every second is not a good solution for me.
Please note I am NOT talking about system monitoring which is served by CloudWatch etc. more for the applications and servers running as well as writing custom application logs. Also all systems are assumed to be Redhat Linux.
Any help or direction is appreciated.
The simplest, easy solution is probably syslogging to a central logserver, e.g. with syslogd. It's simple, free, and widely used in production environments.
You could then use Graylog2 as a free log analyzer. It's a bit like Splunk, but much more affordable.
Also interesting for your toolbox is logstash, a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). You can use the built-in web interface or search the logs via Graylog2, if you prefer its interface. Logstash helps you take logs and other event data from your systems, normalize them and move it into a central place.