Logstash kv field split problem

Elastic Stack Logstash
#1

Logs like this:

topic = abc, partition = 2, offset = 4386, serialized key size = -1, serialized value size = 1139, key = null, value = {"Name":"pc-1","test":false,"message":"elk test - Your pc-1 better. Thank you.","Count":1,"valid":true,"includeResend":false,}

My logstash config file looks like below:

kv {
value_split => ","
value_split => "="
allow_duplicate_values => false
}

Fields look like below:
topic = abc
partition = 2
offset = 4386
serialized key size = -1
serialized value size = 1139
key = null
value = {"Name":"pc-1"

But I need fields below:

topic = abc
partition = 2
offset = 4386
serialized key size = -1
serialized value size = 1139
key = null
value = {"Name":"pc-1","test":false,"message":"elk test - Your pc-1 better. Thank you.","Count":1,"valid":true,"includeResend":false,}

So I wish help to solve this. Thanks.

#2

Better you use Dissect to archive what you want.
Below thew snippet:

"dissect": {
        "mapping" => {
        "message" => "%{*topic} = %{&topic}, %{*partition} = %{&partition},  %{*offset} = %{&offset}, %{*serialized_key->} = %{&serialized_key}, %{*serialized_value->} = %{&serialized_value}, %{*key} = %{&key}, %{*value} = %{&value->}"
         }
  }
#3

You could also use grok to pull out [value], then mutate+gsub to remove it before running the message through the kv filter.

grok { match => { "message" => "value = %{DATA:value}$" } }
mutate { gsub => [ "message", "value = .*$", "" ] }
kv ...
#4

@Badger
Which one is fastest? Grok or dissect?
Need your comment to make my parsers work more efficient... :slightly_smiling_face:

#5

No way to know for sure without benchmarking it. In general dissect has much less functionality so it tends to be cheaper. If grok has to do a lot of back-tracking it can be really expensive (hence the default 30 second time limit to match a pattern), but I would expect a pattern that is anchored at the beginning (by "value =") and then captures the rest of the string to be cheaper than most groks.

#6

Here is an article that give an idea

2 Likes
#7

Thanks @ylasri.

Regards,
Fadjar

#8

Wow its working better. Thanks all for helping me.

1 Like
#9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.