Home / server / Questions / 751126
In Process
The Digital Ninja
Asked: 2016-01-23 10:32:34 +0800 CST

Graylog extractor for comma separated key value pairs

  • 772

I think I'm just not understanding or missing a core concept of graylog and its extractors. I just want to take my key value pairs that are comma delimited and break them out into respective fields.

Sample Log Message

2016-01-22 18:04:05,639 - host_info_log - INFO - ' cpu_count=2,user_cpu=0.0,system_cpu=0.0,idle_cpu=100.0,total_memory=3955.07,avail_memory=3717.3,percent_memory=6.0,used_memory=523.44,free_memory=3431.63,active_memory=378.54,inactive_memory=67.38,swap_memory_used=0.0,swap_memory_total=0.0,swap_memory_free=0.0,swap_memory_percent=0.0

Note: keys are not always in the exact same location, most of the time cpu_count is first, but not always.

graylog

1 Answers

  1. The key (ha!) is to add a converter for Key=Value pairs to fields.

    Using both CSV and Key=Value converters doesn't do what you want however, neither know about the other delimiter. Key=Value assumes whitespace. So one solution is to use a Replace with regular expression extractor to make comma into space, and add Key=Value at the end (remember to hit the Add button).

    Add a string condition for performance reasons and to avoid incorrect extractions.

    Final result looks something like this:

    {
  "extractors": [
     {
      "condition_type": "string",
      "condition_value": "host_info_log",
      "converters": [
        {
          "type": "numeric",
          "config": {}
        },
        {
          "type": "tokenizer",
          "config": {}
        }
      ],
      "cursor_strategy": "copy",
      "extractor_config": {
        "regex": ",",
        "replacement": " ",
        "replace_all": true
      },
      "extractor_type": "regex_replace",
      "order": 0,
      "source_field": "message",
      "target_field": "host_info_log",
      "title": "serverfault  http://serverfault.com/q/751126/241174"
    }
  ],
  "version": "1.3.3 (0fda9dc)"
}
    • 3