Using KV and Grok to parse complex data

Elastic Stack Logstash
1

Hi,

My data is structured like this-

INFO {"datetime": "2021-06-1 22:13:29.469000", data:{"val1":3.14, "val2": 2.17}}
INFO {"datetime": "2021-06-1 21:14:00.469000", data:{"val3":9}}

I want to use logstash to process this data to upload to elastic search.

I have 2 issues-

  1. The key inside the 'data' parameter is dynamic. I think I can use KV to fix this. Am I correct?
  2. The number of key-value pairs inside the data parameter can be just 1 or 10. IE, it is dynamic. How do I work with this? Do I nest grok instructions?

I have tried several times and failed to create a filter. Any help will be highly appreciated.

Thanks

2

Your log is not complex, it looks like a string followed by a json document, but it is not a valid json because the data is not inside double quotes, is this a typo or is this how it looks in your message?

Considering that this is how it looks, you can transform it in a valid json and use dissect and the json filter to parse it.

For example, using this as sample messages:

INFO {"datetime": "2021-06-1 22:13:29.469000", data:{"val1":3.14, "val2": 2.17}}
INFO {"datetime": "2021-06-1 21:14:00.469000", data:{"val3":9}}

The following filter will parse those messages:

filter {
    mutate {
        gsub => ["message", 'data:', '"data":']
    }
    dissect {
        mapping => {
            "message" => "%{logLevel} %{jsonData}"
        }
    }
    json {
        source => "jsonData"
    }
}

The mutate filter with gsub will transform the json part in a valid json, that can be parsed with the json filter.

The dissect filter will split your message in two, the first parte will have the log level information, in this case INFO, and the second will have the json data.

The json filter will parse your json, so the end result will be something like this:

{
  "logLevel": "INFO",
  "@timestamp": "2021-06-02T15:06:34.822Z",
  "datetime": "2021-06-1 22:13:29.469000",
  "data": {
    "val1": 3.14,
    "val2": 2.17
  },
  "jsonData": "{\"datetime\": \"2021-06-1 22:13:29.469000\", \"data\":{\"val1\":3.14, \"val2\": 2.17}}",
  "message": "INFO {\"datetime\": \"2021-06-1 22:13:29.469000\", \"data\":{\"val1\":3.14, \"val2\": 2.17}}"
}
{
  "logLevel": "INFO",
  "@timestamp": "2021-06-02T15:06:34.883Z",
  "datetime": "2021-06-1 21:14:00.469000",
  "data": {
    "val3": 9
  },
  "jsonData": "{\"datetime\": \"2021-06-1 21:14:00.469000\", \"data\":{\"val3\":9}}",
  "message": "INFO {\"datetime\": \"2021-06-1 21:14:00.469000\", \"data\":{\"val3\":9}}"
}
1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.