Home / server / Questions / 963628
Accepted
n.r.
Asked: 2019-04-19 00:11:57 +0800 CST

Line breaks in PHP's stack trace in nginx error logs disturbing logstash analysis

  • 772

I am using nginx with PHP-FPM and ELK as log file analysis.

When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file.

Problem is: Sometimes those error logs contains line breaks which logstash cannot handle, because line breaks are considered as a new log line. 

2019/04/17 19:23:00 [error] 8356#8356: *4403 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function wp_using_themes() in /htdocs/wp-includes/template-loader.php:7
Stack trace:
#0 /htdocs/wp-blog-header.php(19): require_once()
#1 /htdocs/index.php(17): require('/htdocs/wp-blog...')
#2 {main}
  thrown in /htdocs/wp-includes/template-loader.php on line 7" while reading response header from upstream, client: 123.123.123.123, server: foobar.de, request: "GET /2014/11/foobar/ HTTP/1.1", upstream: "fastcgi://unix:/run/php/php-fpm-foobar.sock:", host: "foobar.de"

How to I either handle those linebreaks with logstash or format those error messages to remove the line breaks?

nginx

2 Answers

  1. Best Answer

    Kudos to @USD-Matt thanks to his hint I now know where to look at and that's the solution:

    There is a multiline feature in the ELK stack. But as I am using the filebeat module to postprocess the logs, I cannot just activate the multiline feature in logstash as mentioned above.

    I have to enable it in filebeat itself (/etc/filebeat/filebeat.yml) as described here: https://www.elastic.co/guide/en/beats/filebeat/master/multiline-examples.html

    But heads up: I am also using the filebeat module "nginx" which claims to handle the multiline issue itself. It doesn't. You can force it to do so by adding the above multiline settings to the module config file: /etc/filebeat/modules.d/nginx.yml

    So I did, and that's how the particular part looks like now:

      error:
    enabled: true
    input:
      multiline.pattern: '^\d{4}\/\d{2}\/\d{2}'
      multiline.negate: true
      multiline.match: after

    var.paths:
      - /var/nginx/foobar_de/logs/error.log*
    • 2

  2. I've not used ELK for a while, but you'll need to modify the config to support multiline log entries. The following is an example of using the multiline codec to merge together any lines that don't start with a date, taken from the official documentation.

    input {
  file {
    path => "/var/log/someapp.log"
    codec => multiline {
      # Grok pattern names are valid! :)
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => "previous"
    }
  }
}

    Note that I'm not sure if your log file is using ISO8601 dates as in the example without actually researching that format, so you may not be able to use this example as-is, but it shows the basics of handling multiline logs entries where the date signifies a new entry.

    https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

    • 1