How to trim zero or more leading and trailing spaces from key-value pairs?

Elastic Stack Logstash
1

I have used kv filter to extract key-value pairs using field_split and have used default_keys to provide default keys in case no value is present corresponding to that field.I have used strip of mutate filter to trim all the spaces from the values of the fields.
But for the same field I am getting two different fields one with the default values(that I mentioned using default_keys) and another field with spaces.

I have tried different regex patterns for one or more spaces to be used in trim_key like

"\s", "\s*" , "[ ]{1,}

but nothing worked
My kv filter looks like:--

kv {
     source => "kvpairs"
     default_keys => ["loc","0",
                      "time","null",
                      "action","null",
                      "orig","null"]
     field_split => "|"
     trim_key => "[ ]{1,}"
     
  }

My log data looks like:---

loc=1810756|time= 23Jan2018 12:16:52|action=accept|orig=11.12.13.14|
loc  =|time= 23Jan2018 12:16:52|action=accept|orig=11.12.13.14|
loc= |time= 23Jan2018 12:16:52|action=accept|orig=11.12.13.14|
    loc    =   2    |  time    =    23Jan2018 12:16:52     |     action    =    accept   |orig=11.12.13.14|
loc=3|time= 23Jan2018 12:16:52|action=accept|orig=11.12.13.14|
loc=7|time= 23Jan2018 12:16:52|    action    = |orig=11.12.13.14|
loc=8|time= 23Jan2018 12:16:52|  action    =     |orig=11.12.13.14|
loc=9|time= 23Jan2018 12:16:52|    action  =|orig=11.12.13.14|
loc=10|time= 23Jan2018 12:16:52|action=accept|   orig   =  11.12.13.14   |
loc=11|time= 23Jan2018 12:16:52|action=accept|orig= |
loc=12|time= 23Jan2018 12:16:52|action=accept|orig=      |
loc=13|time= 23Jan2018 12:16:52|    action    =    accept    |orig=|
loc=14|time= 23Jan2018 12:16:52|   action   =   accept  |orig=11.12.13.14|
loc=15|time= 23Jan2018 12:16:52|action=accept|orig=11.12.13.14|
loc=16|time= 23Jan2018 12:16:52|action=accept|orig=11.12.13.14|

The output that I should I get in

Case1:When value is not present,e.g action =| 

should be -> "action" = "null"

Case2:When one or more spaces are present,e.g.     action   =      |    

should be -> "action" = "   "
2

Why not use a mutate gsub filter first to 'fix' the field. Something like this might work:

input {
  generator {
    lines => ['   a    =   23  |   b =    spaced string   ']
    count => 1
  } 
} 

filter {
  mutate {
    gsub => [
      "message", "^\s*", "",
      "message", "\s*$", "",
      "message", "\s*=\s*", "=",
      "message", "\s*\|\s*", "|"
    ]
  }
  kv {
    field_split => "|"
  }
}

output {
  stdout { codec => rubydebug }
}
3

So this will only remove the leading and trailing spaces in the key value pairs.Am I correct??
And can you please explain me the gsub filter part.
Thanks in advance

4

The first two remove leading and trailing spaces in the field. The last two remove spaces around the separators.

5

Thanks that helped a lot.
Any idea why is trim_key not working?

6

I don't know. Might be a bug.

7

@Christian_Dahlqvist the gsub regex is not removing spaces around =.

shot

8

Oh I got the mistake

"message", "\s*=\s*/", "=",

should be:---

input {
generator {
lines => ['   a    =   23  |   b =    spaced string   ']
count => 1
} 
} 

filter {
mutate {
gsub => [
  "message", "^\s*", "",
  "message", "\s*$", "",
  "message", "\s*=\s*", "=",
  "message", "\s*\|\s*", "|"
]
}
kv {
field_split => "|"
}
}

output {
stdout { codec => rubydebug }
}
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.