Home / server / Questions / 641290
In Process
BDM
Asked: 2014-11-01 21:27:58 +0800 CST

Indexing individual cookies with Logstash

  • 772

I'm working on some analysis of Apache logs. Our logs happen to capture the cookies the browser communicates it has and we log it with the request. Right now I can import that long string fine, however I'm wondering if there is a way to have an variable length array per line that lets me search for or perform corollations against individual cookies.

So, based on this message line

"message" => "x.x.x.x - - [30/Oct/2014:20:05:01 -0400] \"GET /js/plugins/backbone/backbone.modelbinding.min.js HTTP/1.1\" 304 - \"https://www.website.com/view/home\" \"Mozilla/4.0 ( compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET4.0C; .NET4.0E; .NET CLR 3.5.30729; .NET CLR 3.0.30729)\" \"AUSERID=5286; SERVICEID=08130xx; JSESSIONID=xxxx; AgencyUserSession=xxxx; _ga=GA1.2.xxx; __utma=xxx ;" + 304/304 1146/213/0 29420/47188900045680 0/5011 -/-/- \"x.x.x.x|-|-|x.x.x.x\"",

I can parse out the cookie and put it in it's own field in it's entirety. However is there a way for me to break up the key/value pairs and store them in an array associated with the log line entry?

["auserid" = "5286", "serviceid" = "08130xx",... etc.

Thank you.

logstash

1 Answers

  1. You can use logstash kv filter to parse cookies.

    If you have a field cookie that containg "AUSERID=5286; SERVICEID=08130xx; JSESSIONID=xxxx; AgencyUserSession=xxxx; _ga=GA1.2.xxx; __utma=xxx;"

    you can write in your logstash.conf something like:

    kv {
    source => "cookie"
    prefix => "cookie_"
    trim => ";"
}

    This will give you fields:

    cookie_AUSERID => 5286
cookie_SERVICEID => 08130xx
...

    But I prefer explicitly specify cookies that I want to be converted to fields (some systems create a lot of cookies that has no meaning for me, they only pollute the logs):

    kv {
    source => "cookie"
    prefix => "cookie_"
    trim => ";"
    include_keys => [ "AUSERID", "SERVICEID" ]
}
    • 0