Working with key:value arrays

Good day !

Accordingly to my logstash configuration I get a message with an array consists of key:value pairs.

I have to mutate it on a very special way adding two fields and put there the key and the value . For example

An incoming message looks like:

"22.114":"1234,88","22.456":"345"

So the mutated one should look as

"Field1":"22.114","Field2":"1234,88"
"Field1":"22.456","Field2":"345"

I guess that I should use ruby but it is unclear to me how to use it to get the result.
Any help would be appreciated.

Thanks

It is unclear what you want to do. Do you want two events each containing [Field1] and [Field2]?

I have an event looks like an array of key:value pairs . I want to transform it into multiple events ,where each of them consists of two fields . A first field value is the key from array and a second field value is the value from array.
Sure , based on my example I want to transform the array into 2 events with those 2 fields

Your values contain commas, so a simple mutate+split will incorrectly parse the [message]. Luckily, a kv filter parses it correctly

kv { field_split => "," value_split => ":" trim_key => '"' target => "[@metadata][kvData]" }
    ruby {
        code => '
            kvData = event.get("[@metadata][kvData]")
            if kvData
                data = []
                kvData.each { |k, v|
                    data << { "Field1" => k, "Field2" => v }
                }
                event.set("someField", data)
            end
        '
    }
    split { field => "someField" }

You can move the results to the top level using this code.

I think using a kv filter is fragile, and I would do it in ruby.

    ruby {
        code => '
            matches = event.get("message").scan(/"([^"]*)":"([^"]*)"(,|$)/)
            data = []
            matches.each_index { |x|
                data << { "Field1" => matches[x][0], "Field2" => matches[x][1] }
            }
            event.set("someField", data)
        '
    }
    split { field => "someField" }

Although that regexp may not be any less fragile than the kv filter :smiley:

1 Like

Thanks a lot ! It is the thing I have been looking for !

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.