we have a few web servers and am planning to create a dashboard to show the real time stats ip address,geo-location and other custom data based on database lookups. Splunk sort of fits perfectly but wondering if there are any open source alternative . i have looked at logstash and graylog2, but to my knowledge they are more of a log analysis tools. Piwik is sort of interesting except that i cannot put any javascript on the webpages. All i have access to is apache web log. Any recommendations please..
I am following the steps in this blog to set up rsyslog + logstash + graylog2 and I can't figure out how to replace the @source_host attribute in logstash using the mutate -> replace filter.
In the exmaple the author replaces his @source_host with a string value but I'd like to use the actual value that is parsed from in this case a syslog.
mutate {
type => loc1
replace => ["@source_host", "loc1"]
}
mutate {
type => loc2
replace => ["@source_host", "loc2"]
}
How do I actually maintain the original source host in my logs?
I was just reading the GELF spec as I'm integrating it into our system. I was just wondering if an additional/custom field can contain a JSON object as the spec doesn't mention it:
So my GELF output (in particular the _request field below) would look something like:
{
"version": "1.0",
"host": "www1",
"short_message": "Short message",
"full_message": "Backtrace here\n\nmore stuff",
"timestamp": 1291899928,
"level": 1,
"facility": "payment-backend",
"file": "/var/www/somefile.rb",
"line": 356,
"_request": {
"ip": "123.123.123.123",
"url": "example.com/dir/file.ext",
"method": "get",
"referer" "example.net"
}
I'm just setting up graylog2 (which is awesome) to be a syslog server for my virtualised environment. All my hosts and switches are happily logging away to graylog2, which is drawing some pretty graphs. So far, so good.
However, when it comes to fowarding the logs from my vSphere 5.0 hosts, I'm running into some issues. I configure the global syslog setting to have a remote host of udp://loghost:514 and I go to the graylog console to see what messages come in. For some reason, graylog2 logs the log severity in the 'host' column:
The screenshot shows some localhost log entries which are correct, and above that some vSphere host entries which obviously are not. All the vsphere entries come in as facility local4 and with a severity of Informational.
The only other time where I had an issue with this was from my Cisco 3750 switch, where I had to explicitly set the syslog forwarding format to syslog, otherwise I'd get all sorts of strange entries in the 'host' column.
Is this a problem with vSphere, or a bug within graylog2? I've not come across anyone else with this problem, so not sure where to start looking.
I have just setup a Graylog2 server and I am looking to send all logs from my main server to the graylog server. I have enabled logging for the main server and am sending logs to my graylog server by adding *.* @logs.example.com:1337 to /etc/rsyslog.conf.
What I want is to have Graylog2 collect all my Apache logs, system logs (for SSH logins, rejected logins) and any other logs I need to monitor.
For the Apache logs, I would also like the Rails logs. My sites are located in /srv/www/ and then the structure is sitename.com/public_html and sitename.com/logs. I have many sites on the server and I would like an easy way to view all of the errors and make some nice graphs out of them hence why I want to use Graylog2...
The log files in the logs folder are access.log and error.log.
The Rails logs would be in sitename.com/public_html/log. This contains production.log.