Split document into multiple documents

[ciavam]

Hello,

Through the logstash code I was able to get a result like:

{field0 = value
field1 = [value1_field1; value1_field1]
field2 = [value1_field2; value2_field2]
}

Is it possible through logstash to go to what is written below?

{field0 = value
field1 = value1_field1
field2 = value1_field2
}

{field0 = value
field1 = value2_field1
field2 = value2_field2
}

Thanks for those who want to help me.

[Badger]

Take a look at the split filter.

[ciavam]

Thanks for your reply @Badger. I have already seen "split". In my configuration file I tried to write:

split {
        field => "field"
        target => "field"
    }

In this way, however, it is possible to divide the document only for a field. Is there a way to do this, but with multiple fields?

[Badger]

@ciavam

Is there a way to do this, but with multiple fields?

Use multiple split filters.

[ciavam]

My goal is to have a match between the i value of the first field and the i value of the second field (as seen in my first post). Is this goal achieved with multiple split filters?

[Badger]

If you start with

{ "field0": "z", "field1": [ "a", "b" ], "field2": [ "c", "d" ] }

then

split { field => "field1" }
split { field => "field2" }

will produce four events, one for each pairing: a/c, a/d, b/c, b/d.

If you want the two events then you could do it in ruby. Your sample data has three value1 and only one value2, so this may be the wrong pairing (this results in a/c, b/d)

    ruby {
        code => '
            f1 = event.get("field1")
            f2 = event.get("field2")
            if f1.is_a? Array and f2.is_a? Array and f1.length == f2.length
                a = []
                f1.each_index { |x|
                    a << { "field1" => f1[x], "field2" => f2[x] }
                }
                event.set("[@metadata][stuff]", a)
            end
        '
    }
    if [@metadata][stuff] {
        split { field => "[@metadata][stuff]" }
        mutate {
            replace => {
                "field1" => "%{[@metadata][stuff][field1]}"
                "field2" => "%{[@metadata][stuff][field2]}"
            }
        }
    }

[ciavam]

Thanks again for your help. Actually my document has more than two list fields (precisely 7), so starting with:

{ "field0": "z", "field1": [ "a", "b" ], "field2": [ "c", "d" ], "field3": ["e", "f"] ... }

I should get to:

{ "field0" : "z",
"field1": "a",
"field2": "c",
"field3": "e"
...
 }

{ "field0": "z",
"field1": "b",
"field2": "d",
"field3": "f"
...
 }

Is your ruby code easily adaptable to the case of more than two list fields?

[Badger]

Sure you can do

        f1 = event.get("field1")
        f2 = event.get("field2")
        f3 = event.get("field3")
        f4 = event.get("field4")
        f5 = event.get("field5")
        f6 = event.get("field6")
        f7 = event.get("field7")

        if f1.is_a? Array and f2.is_a? Array and f3.is_a? Array and f4.is_a? Array and f5.is_a? Array and f6.is_a? Array and f7.is_a? Array and f1.length == f2.length and f1.length == f3.length and f1.length == f4.length ...
            a = []
            f1.each_index { |x|
                a << {
                    "field1" => f1[x], "field2" => f2[x], "field3" => f3[x], 
                    "field4" => f4[x], "field5" => f5[x], "field6" => f6[x], 
                    "field7" => f7[x] 
                }
            }
            event.set("[@metadata][stuff]", a)
        end

[ciavam]

Thank you so much @Badger , it works! The only problem is that:

"field" => "{ \ "field \ " : numberValue}"

rather than:

"field" => numberValue

Unfortunately I'm new to logstash. Sorry for too many questions, could you help me last time?

[Badger]

You are still using the split and doing the mutate for all seven fields, right?

[ciavam]

Yes, right.

[Badger]

@ciavam

"field" => "{ \ "field \ " : numberValue}"

OK, so use another mutate to

mutate { replace => { "field" => "%{[field][field]}" } }

[ciavam]

Thanks again