KV how to extract data between square brackets

Assume I have the following message:

"Some message [key1=val1] more text [key2=some [special] value], more text."

From the above message I would like to extract key1 and key2 as follows:
key1 = val1
key2 = some [special] value

How can I achieve that? I found only 1 topic similar to this:

Tried to play around with the solution given there but it never seems to be working for the case above.

A more realistic example:
Found cached [valuejson={"liveStreams":[]}] for [controller=Game] and [action=GetLiveStreams].

I want to extract:

valuejson = {"liveStreams":[]}
controller = Game
action = GetLiveStreams

How do you know that it shouldn't be

key1 = val1] more text [key2=some [special] value

That's a serious question. What is a definition of the pattern that matches the RHS?

Hi @Badger

If i would have to describe it in words i'd say there are 2 options to deal with what I want to achieve:

  1. match every nearest pair of square brackets that contains = inside them (so if there's square brackets without = inside them, they will be used as a part of the value).
  2. match fields only if there's an equal amount of square brackets: https://stackoverflow.com/questions/546433/regular-expression-to-match-balanced-parentheses

I tried to do as in that stackoverflow topic, unfortunately it seems like logstash fails to read the regexes that are suggested in the answer there, logstash doesn't even load.

Edit: here is an example from that topic that seems to work when I test it, but doesn't work in logstash itself:

So after I understood that recursive regex isn't supported probably I tried a different regex with max 2 levels of nesting:

It seems to work when testing:

Logstash also loads successfully, but still splits the fields in the wrong way...

What am I missing? perhaps need a different filter for this one? or even ruby code?
Thanks for the help.

Anyone any ideas? ;o

@Badger can the pattern you suggested on the link I gave in the first post be modified somehow to match my needs?

I do not know enough about regexps to write one that does what you want.

@Badger can I perhaps use Ruby code to achieve this?
For example, I already wrote a regex to capture the groups I need here: KV how to extract data between square brackets
Can I write a ruby code to save them under a certain event, and then send this event to KV filter?

For these interested, here's how I achieved it:

ruby {
	code => 'event.set("kv", event.get("tempMessage").scan(/\[(?:[^\]\[]+|\[(?:[^\]\[]+|\[[^\]\[]*\])*\])*\]/))'
kv {
	source => "kv"
	field_split_pattern => "(?:^\[|\]$)"
	trim_key => " "
	trim_value => " "

Apparently kv can also take parameter of array of strings, which makes life lot easier for my case since it treats each array value independently.

Thanks for the helpers.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.