Home / server / Questions / 896103
Accepted
trueCamelType
Asked: 2018-02-08 06:08:53 +0800 CST

Why is this exclude_lines in filebeat excluding all logs?

  • 772

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-access log. This happens every second, and I'd like to ignore it. My regex matches these lines in the regex testers I'm using, but it appears to have stopped all logs coming from that file, instead of the expected single lines.

filebeat:
  prospectors:
    paths:
      - /var/log/gitlab/nginx/gitlab_access.log
    input_type: log
    exclude_lines: ['(.*\bPUT\b)(.*\bgitlab-ci-multi-runner).*']
    document_type: gitlab_access

Below is an example of the log file, and I want to block every line that is a PUT from the gitlab-ci-multi-runner.

**192.168.1.105 - - [07/Feb/2018:07:53:36] "PUT /ci/api/v1/builds/1738.json HTTP/1.1" 404 3082 "" "gitlab-ci-multi-runner 1.3.0 (1-6-stable; go1.3.3; linux/amd64)"**
192.168.1.110 - - [07/Feb/2018:07:53:37] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.2.0 (7-2-stable; go1.7.5; windows/amd64)"
**192.168.1.105 - - [07/Feb/2018:07:53:39] "PUT /ci/api/v1/builds/1738.json HTTP/1.1" 404 3082 "" "gitlab-ci-multi-runner 1.3.0 (1-6-stable; go1.3.3; linux/amd64)"**
192.168.1.110 - - [07/Feb/2018:07:53:40] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.2.0 (7-2-stable; go1.7.5; windows/amd64)"
192.168.1.55 - - [07/Feb/2018:07:53:40] "GET / HTTP/2.0" 200 45895 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36"
192.168.1.55 - - [07/Feb/2018:07:53:41] "GET /assets/favicon-075eba763121a0c1f89a89ee81678bcde72e2a47cd3a42.ico HTTP/2.0" 200 1384 "https://gitlab.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.78 Safari/537"
192.168.1.55 - - [07/Feb/2018:07:53:41] "GET /uploads/-/system/user/avatar/21/yodaProfile.jpg HTTP/2.0" 304 0 "https://gitlab.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36"
**192.168.1.105 - - [07/Feb/2018:07:53:42] "PUT /ci/api/v1/builds/1738.json HTTP/1.1" 404 3082 "" "gitlab-ci-multi-runner 1.3.0 (1-6-stable; go1.3.3; linux/amd64)"**
192.168.1.110 - - [07/Feb/2018:07:53:43] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.2.0 (7-2-stable; go1.7.5; windows/amd64)"

I expected the lines that start with ** to be removed, and the rest to go through, but this isn't happening (none of these lines make it through now). If I remove the regex, everything comes through again.

elasticsearch

1 Answers

  1. Best Answer

    If you simplify your exclude_lines-configuration to the following, it will be matched by filebeat.

    exclude_lines: ['\"PUT.*gitlab-ci-multi-runner']

    I have read through the exclude_lines and the regexp-support documentation, but I didn't figure out the reason why your initial regexp does not match the three lines, since they match when I add it to regexr.com and choose PCRE as the regex engine.

    If you want to find out what caused it to not be matched I would suggest that you remove one and one element from the regexp until it matches.

    First remove the grouping

    exclude_lines: ['.*\bPUT\b.*\bgitlab-ci-multi-runner.*']

    Then try to remove the \b entries

    exclude_lines: ['.*PUT.*gitlab-ci-multi-runner.*']

    Then you should get to something similar to my answer.

    exclude_lines: ['PUT.*gitlab-ci-multi-runner']

    You can also remove one and one entry at a time, and not all \b elements. When you figure out which entry caused exclude_lines to not match, it will be much easier to find out why.

    I hope this answer will help you along the way!

    • 3