Questions[grok](server)

I have Filebeat-7.1 installed in a Debian server, this Filebeat send data from files in this Debian server to server with Logstash 7.6 , here are the files config

Filebeat.yml:

#=========================== Filebeat inputs =============================

filebeat.inputs:

type: log

Change to true to enable this input configuration.
enabled: true
paths:

/root/code/cigol/logs/server.log
json.keys_under_root: true
json.overwrite_keys: true
json.add_error_key: true
force_close_files: true
fields:
env: dev
type: voiceserver.log

type: log
enabled: true
paths:
- /usr/local/freeswitch/log/freeswitch.log
force_close_files: true
fields:
env: dev
type: freeswitch.log

processors:

drop_fields:
fields: ["agent.ephemeral_id", "time", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset", "@version", "fields.env", "tags"]
#----------------------------- Logstash output --------------------------------
output.logstash:

hosts: ["35.171.202.75:5044"]

--------------------------------logstash.conf----------------------------------------------------------------------------- input.conf

input {
beats {
port => 5044
}
}

filter.conf

filter{
if [fields][env] == "dev" {
if [source] == "/root/code/cigol/logs/server.log" {
json {
source => "message"
}
}
} else
if [source] == "/usr/local/freeswitch/log/freeswitch.log" {
grok {
match => { "message" => "%{NOTSPACE:uuid} %{TIMESTAMP_ISO8601:date} [%{LOGLEVEL:loglevel}] %{GREEDYDATA:message}" }
remove_field => ["message"]
}
}
}

Output.conf

output {

elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "%{[fields][type]}-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

application logs format

79110982-6d35-4b80-9be7-6ec9772313f9 2020-04-21 14:25:55.001130 [DEBUG] switch_core_state_machine.c:749 (sofia/3clogic_external/[email protected]:5505) State DESTROY

Kibana Output

message 79110982-6d35-4b80-9be7-6ec9772313f9 2020-04-21 14:25:55.001130 [DEBUG] mod_sofia.c:364 sofia/3clogic_external/[email protected]:5505 SOFIA DESTROY

I want to segregate message as below

"UUID" = 79110982-6d35-4b80-9be7-6ec9772313f9
"date" = 2020-04-21 14:25:55.001130
"loglevel" = DEBUG
"message" = switch_core_state_machine.c:749 (sofia/3clogic_external/[email protected]:5505) State DESTROY

The logstash documentation indicates that you can collapse the multiple indented lines in a Java stacktrace log entry into a single event using the multiline codec:

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

input {
   syslog {
       type => syslog
       port => 8514
       codec => multiline {
            pattern => "^\s"
            what => "previous"
       }
  }
}

This is based on logstash finding an indent at the start of the line and combining that with the previous line.

However, the logstash documentation is the only place where I can find a reference to this. The general user community seems to be using elaborate grok filters to achieve the same effect.

I've tried the basic indentation pattern provided by logstash, but it doesn't work. Has anyone else managed to get this working by matching the indentation pattern?

I'm looking through some syslog logs files in my ELK stack and noticed that all the syslog_severity fields are 'notice', when I can verify in the log files that they are not 'notice'. Seems like Logstash is defaulting syslog_severity to notice. Whenever I go to kibana to search, no matter what I put in the logstash filter configuration file, kibana always returns notice for the severity. I have this in my logstash filter configuration:

filter {
 if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOG5424LINE}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri {
      syslog_pri_field_name => "syslog5424_pri"
    }
    date {
      match => [ "syslog5424_ts", "ISO8601" ]
    }
  }
}

I've looked at the solution mentioned here but don't think that applies in my case, if you look at my filter config file. I've tried the solution mentioned here and restarted my logstash service with

sudo service logstash restart          

I've also tried restarting the rest of the services in my ELK stack, still getting notice for all of my syslog_severity fields. Any idea what needs to be changed in the filter?

My log messages are of this format:

<134>1 2015-01-01T11:12:23.180242-02:00 message

I've tested this grok with my messages in the grok debugger and it parses my messages just fine.

I need to match this entry

2015/10/30 23:58:21 pid 22223 [email protected] 192.168.0.1 [p4/2012.2/LINUX26X86_64/536738] 'test-monitor show'

To match this I wrote this Regex

P4_DATE (?>\d\d){1,2}\/(?:0[1-9]|1[0-2])\/(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
P4_TIME (?:2[0123]|[01]?[0-9])\:(?:[0-5][0-9])\:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
P4_PID \b(?:[1-9][0-9]*)\b
P4_USER \b\w+\b
P4_HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
P4_IP (?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])

and then overall

 P4_MATCH %(P4_DATE:p4date} %{P4_TIME:p4time} pid %{P4_PID:p4pid} %{P4_USER:p4user}\@%{P4_HOSTNAME:p4client} %{P4_IP:p4remoteclient} [%{DATA:p4version}] \'%{DATA:p4action}\'" }

and then match via

match => [ "message", "%{P4_MATCH}" ]   

But still getting "_beats_input_codec_plain_applied, _grokparsefailure"

I am not a regular expression expert but any help is really appreciated.

I'm sure there is a simple solution to this but I'm new to working Logstash. The filter I'm trying to apply is for some firewall logs and I have a field that will either have a single or two values separated by a colon.

X16-V523 X16-V523:example.com

I have been using for the line with two values,

%{GREEDYDATA:srcint}:%{GREEDYDATA:srchost}

How do i make the second part of the match optional?

Chris