KV Filter not extracting data

[vivek14]

After using grok and mutate filters, I have a field called headers containing following value

accept-encoding=[gzip,deflate], breadcrumbId=[ID-fuse1-51739-1547625048671-0-6180], Host=[localhost:9000], JMSCorrelationID=[Camel-ID-fuse3-56675-1547627177108-9-21], User-Agent=[Apache-HttpClient/4.1.1 (java 1.5)], wsErrorCode=[WS002], wsErrorMsg=[ESB Connection Timeout]

I want to extract the value from the key breadcrumbId. i.e. I want ID-fuse1-51739-1547625048671-0-6180 and put it into the field UniqueProcessId.

I am using following syntax

Blockquote

kv {
source => "headers"
field_split => ","
include_brackets => "true"
remove_char_value => ","
include_keys => [ "breadcrumbId"]
target => "UniqueProcessId"
}

Blockquote

The breadcrumId's value is not being set to the target. The UniqueProcessId's value is blank. There is no parsing error.

I tried with and without target options. Both not working.

Am I using the KV filter correctly? Is it because one of the key's value contains a ,? accept-encoding=[gzip,deflate]

Appreciate any help.

Thanks

[Badger]

Either include the leading space in the key name

include_keys => [ " breadcrumbId"]

which gets you

"UniqueProcessId" => {
    " breadcrumbId" => "ID-fuse1-51739-1547625048671-0-6180"
}

Or have the kv filter remove it using

remove_char_key => " "

Personally I would not use

remove_char_value => ","

[vivek14]

Thank you @Badger.

Since I do not have any control on the contents of the headers field, I tried using the 2nd option that you suggested i.e. to use remove_char_key => " ". It works.

However, the value I get is as shown below

{
"breadcrumbId": "ID-fuse2-62536-1547626076191-0-25930"
}

Shouldn't the ouput be like this ?

ID-fuse2-62536-1547626076191-0-25930

How is KV filter supposed to work?

How do I get just the value part ID-fuse2-62536-1547626076191-0-25930 from the kv filter?
Thanks,

[Badger]

If you are questioning why you get breadcrumbId inside UniqueProcessId, that's just the way kv works. If it was extracting multiple keys it would put them all inside the target.

You could try not specifying target, then doing a mutate+rename to change the field name.

[vivek14]

Thank you @Badger.

I did as you advised. It meets my requirement.

I am new to Elastic Stack and learning on the job. Still long way to go.

Thanks for your help.