Home / server / Questions / 707784
Accepted
Olly
Asked: 2015-07-23 14:04:16 +0800 CST

logstash failing to parse syslog input

  • 772

I've configured logstash (v1.5.0), with a simple syslog input, as follows:

input {
  syslog {
    type => syslog
    port => 5514
  }
}

filter {
  kv {}
}

output {
  elasticsearch {
    cluster => "logs"
    host => "0.0.0.0"
    protocol => "transport"
  }
}

However it seems to be failing on some of the cron logs. The following line fails to parse with a _grokparsefailure_sysloginput:

<77>Jul 22 22:01:01 ip-172-31-2-48 run-parts(/etc/cron.hourly)[2599 finished 0yum-hourly.cron

The final JSON output is:

{
  "_index": "logstash-2015.07.22",
  "_type": "syslog",
  "_id": "AU63yLrC118PBgBqQxRA",
  "_score": null,
  "_source": {
    "message": "<77>Jul 22 22:01:01 ip-172-31-2-48 run-parts(/etc/cron.hourly)[2599 finished 0yum-hourly.cron\n",
    "@version": "1",
    "@timestamp": "2015-07-22T22:01:01.569Z",
    "type": "syslog",
    "host": "172.31.2.48",
    "tags": [
      "_grokparsefailure_sysloginput"
    ],
    "priority": 0,
    "severity": 0,
    "facility": 0,
    "facility_label": "kernel",
    "severity_label": "Emergency"
  },
  "fields": {
    "@timestamp": [
      1437602461569
    ]
  },
  "sort": [
    1437602461569
  ]
}

Any pointers?

logstash

3 Answers

  1. Best Answer

    The syslog input use grok internally, your message is probably not following the syslog standard 100%.

    The solution in this link worked for me: http://kartar.net/2014/09/when-logstash-and-syslog-go-wrong/

    The key info from the link is:

    Replace the existing syslog block in the Logstash configuration with:

    input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }
}

    Next, replace the parsing element of our syslog input plugin using a grok filter plugin.

    filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
  }
}

    You can edit the filter matching ("grok") syntax now, to match your desired format. It's also possible to support multiple different syntaxes with creative use of if, else if, and else.

    • 6

  2. Coming here after 4 years, now the logstash syslog input supports setting the grok pattern to use, as detailed in the documentation.

    In order to keep the syslog input functionalities, one can as such insert the nonstandard pattern to parse in the grok_pattern setting, e.g.:

    input {
  syslog {
    port => 514
    type => "syslog"
    grok_pattern => "(?:<%{POSINT:priority}>%{SYSLOGLINE}|YOUR NONSTANDARD PATTERN HERE)"
   }
}

    or likewise amend the default <%{POSINT:priority}>%{SYSLOGLINE} pattern to make it match also the nonstandard input lines.

    • 3

  3. I've had the same problem on logstash 7.17. Solved it by adding ecs_compatibility => "v8" in syslog input plugin configuration:

    input {
    syslog {
        port => "514"
        ecs_compatibility => "v8"
    }
}
...
    • 0