Kvpairs where one of the values is a JSON structure

Elastic Stack Logstash
1

I have a log that I am parsing using grok and kv:

filter {
  grok {
    match => {"message" => "msg=%{WORD:action} %{GREEDYDATA:kvpairs}" }
  }
  kv {
    source => "kvpairs"
    remove_field => ["kvpairs"]
  }
}

One of the values in my kvpairs is structured JSON data that I want to interpret as additional key/value pairs. How can I do this?

2

You will probably want to do another grok + match.

3

I'm very new to logstash -- could you give me a hint as to how to do that. Would I add another grok filter after the kv filter? If my key is "jsondata" what would that look like. Thanks in advance! (If there is good documentation with examples that you could point me to, I'd be happy to RTFM -- but I'm finding just the basic documentation that defines the format for each of the filters a little inadequate for me to get rolling with.)

4

Yeah exactly! It's exactly the same as what you have but do it on the other KV pair you want to split.

5

Thanks! I tried this and it almost worked but it looks like kv truncated my json string, so I got a json parse error:

filter {
  grok {
    match => {"message" => "msg=%{WORD:action} %{GREEDYDATA:kvpairs}" }
  }
  kv {
    source => "kvpairs"
    remove_field => ["kvpairs"]
  }
  grok {
    match => {"jsondata" => "%{GREEDYDATA:jsonpairs}" }
  }
  json {
    source => "jsonpairs"
    remove_field => ["jsonpairs"]
  }
}

Is there a limit on the length of the value in a kv pair? If so, how can I increase it?

6

I don't think so, but what was the error you got?

7

I'm looking at the log entry in Kibana and I see a

tags: _jsonparsefailure

entry in the log

And I can see that the jsonpairs string is truncated (as is jsondata). I can see the full json string in the message value.

8 
grok {
  match => {"jsondata" => "%{GREEDYDATA:jsonpairs}" }
}

This filter serves no purpose. It's just a convoluted way of copying a field to another.

Where did the JSON message get truncated? Can you produce a minimal reproducible example? Did it by any chance get truncated near an equals sign or something else that has a special meaning to kv?

9

You're right about the grok of course -- I realized afterwards that jsondata and jsonpairs would always be identical.

The truncation happened right in the middle of a quoted string, not near any special characters, but right after the word "Error" in case that is significant. -- that does seem to be significant because I created an example where the json string is a little longer before that part, and it still truncates right after "Error".