Home / server / Questions / 819182
In Process
Prashant Lakhera
Asked: 2016-12-07 18:04:22 +0800 CST

Logstash Grok Parsing Failed

  • 772

I need to match this entry

2015/10/30 23:58:21 pid 22223 [email protected] 192.168.0.1 [p4/2012.2/LINUX26X86_64/536738] 'test-monitor show'

To match this I wrote this Regex 

P4_DATE (?>\d\d){1,2}\/(?:0[1-9]|1[0-2])\/(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
P4_TIME (?:2[0123]|[01]?[0-9])\:(?:[0-5][0-9])\:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
P4_PID \b(?:[1-9][0-9]*)\b
P4_USER \b\w+\b
P4_HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
P4_IP (?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])

and then overall

 P4_MATCH %(P4_DATE:p4date} %{P4_TIME:p4time} pid %{P4_PID:p4pid} %{P4_USER:p4user}\@%{P4_HOSTNAME:p4client} %{P4_IP:p4remoteclient} [%{DATA:p4version}] \'%{DATA:p4action}\'" }

and then match via 

match => [ "message", "%{P4_MATCH}" ]

But still getting "_beats_input_codec_plain_applied, _grokparsefailure"

I am not a regular expression expert but any help is really appreciated.

regular-expressions logstash grok

3 Answers

  1. I don't see the specific issue, but take a look at https://grokdebug.herokuapp.com/ -- build up your patterns one field at a time and see what step causes matching to fail.

    • 1

  2. Interesting approach. The Grok library that comes with Logstash has some prebuilt patterns to help you avoid having to build large regexes like you're doing there. You may have better luck reusing their engineering than building your own.

    • 0

  3. You have a syntax error in p4date where there is a bracket instead of a curly brace on the field it should be %{P4_DATE:p4date}.

    You need to escape the square brackets for p4version which should be \[%{DATA:p4version}\]

    There are extra characters " } after the p4action which need removing .

    THIS P4_MATCH %(P4_DATE:p4date} %{P4_TIME:p4time} pid %{P4_PID:p4pid} %{P4_USER:p4user}\@%{P4_HOSTNAME:p4client} %{P4_IP:p4remoteclient} [%{DATA:p4version}] \'%{DATA:p4action}\'" }

    SHOULD BE THIS P4_MATCH %{P4_DATE:p4date} %{P4_TIME:p4time} pid %{P4_PID:p4pid} %{P4_USER:p4user}\@%{P4_HOSTNAME:p4client} %{P4_IP:p4remoteclient} \[%{DATA:p4version}\] \'%{DATA:p4action}\'

    • 0