KV filter trim with regex

Elastic Stack Logstash
1

Given the example filter inputs "[Key1:/app/segment/confirm.jsp] 21:50:00 [Key2:3] [Key3:93] [Key4:392] [Key5:210] [Key6:144] [Key7:93] [Key8:392] Some more text that should not be included in key value"

I need to extract the key value pairs between the brackets [ ] , however, can't find a way to trim away the exes characters after ending bracket ] . Number of keys are dynamic so cant use grok to extract what I want.

KV filter looks like this now:
kv{
source => "message"
value_split => ":"
field_split => "["
include_brackets => false
trimkey => " "
}

was looking to add a trim setting with a regex to remove all chars including ] until the next [ , but this does not seem possible.

Any other approaches to take on this problem? Guess I could look to extract alle the key value segments first with a regex like this [(?.?)(?::(?.?))?] but cant see how to do that.

Any help is much appreciated

2

You seem to have a timestamp embedded within your list which could complicate things. Why not use grok to extract a pure key-value list into a separate field and then strip the leading [ and training ]. Once that is done and you have something like Key2:3] [Key3:93] [Key4:392] [Key5:210] [Key6:144] [Key7:93] [Key8:392 , I believe you should be able to define ] [ as the field split.

3

Sounds like a good approach, however I do not know all the keys and the number of key value items are dynamic. Might also be other text strings, like the time stamp, between the items.

Can't wrap my head around how to pull out a dynamic amount of key value items wrapped in [ ] using Grok. Any suggestions?

4

You could try this filter before kv filter :

mutate {
    gsub => ["message", "\] [^\[]+", "] "]
}

It will remove all characters between ] and [

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.