I am using logstash to collect logs from a group of webapps and send them to graylog2 for centralized viewing.
I have the following filter for tokenizing:
grok {
type => "webapps"
pattern => "^%{TIME:timestamp} \[%{NOTSPACE:thread}\]%{SPACE}%{WORD:loglevel}%{SPACE}%{JAVACLASS:class} - %{GREEDYDATA:short_message}"
drop => false
debug => "true"
add_tag => [ "%{loglevel}" ]
}
Loglevel will be along the lines of TRACE, DEBUG, INFO, NOTICE, ERROR, FATAL.
Initially, everything was showing up as "Alert" (which has a numeric value of 5 in ruby/graylog-server and is marked in the sources as "unknown").
I then added a series of mutate filters, such as this ones:
mutate {
type => "webapps"
tags => "INFO"
add_tag => [ "ll_%{@level}", "mutated" ]
replace => [ "@level", "6" ] # informational
}
This got me closer to the proper loglevel/severity showing up in the webinterface, but all the "INFO" messages were showing up as being Debug messages.
I wrote a script to look through the elasticsearch data and set the level field based on what it should be.
- Debug : 7
- Informational : 6
- Warning : 4
- Error : 3
- Critical : 2
with levels 0, 1 and 5 not being used, as the sources indicated these were reserved.
But this script is expensive in terms of resources, and I don't think it would work too well when the dataset it is trying to update is growing at a rate of hundreds or thousands of messages per second.
The other thing I noticed is that when I use something like "@source" in a tag, the proper value shows up. When I do "@level", I get that inserted as a literal, which indicates that it isn't a predefined field has had been indicated.
I've looked at the sources a little, but not exhaustively, so it's possible I've just missed it.
The question is, what do I need to change in my mutate filter to make "INFO" show up as "Informational" in the graylog2 interface?
Nothing I tried worked, and the mailing list wasn't able to help either.
What I wound up doing was to stop using logstash for this, and add in a logback-gelf jar and add to my logback configuration.
It has worked like a charm since then.