KV how to extract data between square brackets

[sahar]

Hello,
Assume I have the following message:

"Some message [key1=val1] more text [key2=some [special] value], more text."

From the above message I would like to extract key1 and key2 as follows:
key1 = val1
key2 = some [special] value

How can I achieve that? I found only 1 topic similar to this:

KV Filtering data between square brackets

Hi, How can i parse data which has two layers of square brackets. Inner square brackets Outer square brackets I only want to consider outer square brackets data as key value pairs. Example : [filename=0_578cc[R2][]2 veckor f=f600re (2).doc] In the above example i want to take filename as key but it is also considering the inner square bracket which is breaking my logic. I am using grok and kv filter with the following config. match => [ "message", "%{SYSLOGTIMESTAMP:eventtime}\s(?[^\s…

Tried to play around with the solution given there but it never seems to be working for the case above.

A more realistic example:
String:
Found cached [valuejson={"liveStreams":[]}] for [controller=Game] and [action=GetLiveStreams].

I want to extract:

valuejson = {"liveStreams":[]}
controller = Game
action = GetLiveStreams

[Badger]

@sahar

From the above message I would like to extract key1 and key2 as follows:
key1 = val1
key2 = some [special] value

How do you know that it shouldn't be

key1 = val1] more text [key2=some [special] value

That's a serious question. What is a definition of the pattern that matches the RHS?

[sahar]

Hi @Badger

If i would have to describe it in words i'd say there are 2 options to deal with what I want to achieve:

1. match every nearest pair of square brackets that contains = inside them (so if there's square brackets without = inside them, they will be used as a part of the value).
2. match fields only if there's an equal amount of square brackets: https://stackoverflow.com/questions/546433/regular-expression-to-match-balanced-parentheses

I tried to do as in that stackoverflow topic, unfortunately it seems like logstash fails to read the regexes that are suggested in the answer there, logstash doesn't even load.

Edit: here is an example from that topic that seems to work when I test it, but doesn't work in logstash itself:

[sahar]

So after I understood that recursive regex isn't supported probably I tried a different regex with max 2 levels of nesting:
\[(?:[^\]\[]+|\[(?:[^\]\[]+|\[[^\]\[]*\])*\])*\]

It seems to work when testing:

Logstash also loads successfully, but still splits the fields in the wrong way...

What am I missing? perhaps need a different filter for this one? or even ruby code?
Thanks for the help.

[sahar]

Anyone any ideas? ;o

@Badger can the pattern you suggested on the link I gave in the first post be modified somehow to match my needs?

[Badger]

I do not know enough about regexps to write one that does what you want.

[sahar]

@Badger can I perhaps use Ruby code to achieve this?
For example, I already wrote a regex to capture the groups I need here: KV how to extract data between square brackets
Can I write a ruby code to save them under a certain event, and then send this event to KV filter?

[sahar]

For these interested, here's how I achieved it:

ruby {
	code => 'event.set("kv", event.get("tempMessage").scan(/\[(?:[^\]\[]+|\[(?:[^\]\[]+|\[[^\]\[]*\])*\])*\]/))'
}
kv {
	source => "kv"
	field_split_pattern => "(?:^\[|\]$)"
	trim_key => " "
	trim_value => " "
}

Apparently kv can also take parameter of array of strings, which makes life lot easier for my case since it treats each array value independently.

Thanks for the helpers.