Kv filter plugin when there are spaces in key and value

Elastic Stack Logstash
1

I am facing challenges in using kv filter plugin. I wanted to split data based on colon : . I have spaces in the key and also value. I wanted to have the data filtered with the key (entire key) with the spaces removed and not remove any space in the value(but can remove the spaces appearing in very first). How can I get that using kv

for eg: if i have data like and wanted to separarate key, value as mentioned below.

         Job id:        this is the JOBID 
         key as "Jobid " 
         value as "this is the JOBID"

I have data like this

I tried following

  1. Option 1

           kv {	trimkey => "\s"
  value_split => ":"
       }

    "message": "Integrated at: Tuesday January 26, 2016 09:40:07 UTC",
    "at": "Tuesday",
    "09": "40:07"

  "message": "Bundles: USE: sft install 2016-01-26-081744.thoma.hs-comp"
  "Bundles": "USE:"

  "message": "Time in queue: 01H 22m 36s"
 "queue": "01H"

  2. Option 2

        kv { value_split => ":"}
   "message": "Integrated at: Tuesday January 26, 2016 09:40:07 UTC",
  "at": "Tuesday",
  "09": "40:07"
2

If you have spaces in your keys and values, I am assuming you are not using space to split fields?
What does a full message look like? How is key-value pairs separated in the record?

3

Thanks Christian for your response
Following is a typical data I have. I am not using the "[split fields] (https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html#plugins-filters-kv-field_split)" and I am splitting the data using value_split => ":"

    message  " Job id:        this is the JOBID : this will the submitted in windows"

I would like to have

     key as "Jobid" 
     value as "this is the JOBID : this will the submitted in windows"
4

So you don't have multiple key-value pairs in the message, the message contains the default field separator and the value contains the character you use to separate key from value? Try setting the field separator to something that does not exist in the key or the value and see if that works. If this is not the case, I don't think this is what the kv filter was designed to parse, so I would look into using either the grok or ruby filters for this.

1 Like
5

Thanks Christian. That really helped.

I gave it as some thing like below and i did the magic

		kv {	trimkey => "\s"
			value_split => ":"
			field_split => "&?&"
		}