Logstash problems with splitting on field value

Elastic Stack Logstash
1

Hi All, i have a problem with the logstash split filter.
Any advice or input is greatly appreciated. I have the input as such:

{"SONG_A":[
{
"MD5":"a",
"filename": "x.txt"
},
{
"MD5":"b",
"filename": "y.txt"
}
]
}

I am trying to split the first key into multiple events. The following code works:

filter{
json{
		source=>"message"
	}
split
	{
	field => "SONG_A"
	}}

Where i am getting the following results:

Event 1:
{
      "SONG_A" => {
        "filename" => "x.txt",
             "MD5" => "a"
    }
}
Event 2:
{
      "SONG_A" => {
        "filename" => "y.txt",
        "MD5" => "b"
    }
}

However, the key - in this case SONG_A is subject to change, as such a grok filter was created and the value was stashed in the metadata field. However, i am unable to split the key into separate events. The code is as below:

filter{
	json{
		source=>"message"
	}
	grok {
	match => {"message" => "(?<[@metadata][SONG]>SONG[_A-Z]*)"}
	}
	split
	{
	field => "[@metadata][SONG]"
	}

However, i am getting the following output where the event is still a single event and not being split as above:

event 1:
{

    "@metadata" => {
        "SONG" => "SONG_A",
    },
     "SONG_A" => [
        [0] {
            "filename" => "x.txt",
                 "MD5" => "a"
        },
        [1] {
            "filename" => "y.txt",
                 "MD5" => "b"
        }
    ]
}

Why am i unable to split the event into two events?

2

[@metadata][SONG] is not an array, it is a string, so a split filter will split it into multiple lines if it contains them. "SONG_A" is a single line, so the filter will be a no-op.

It looks like the field name [SONG_A] is a variable, so you want to do the equivalent of

split { field => "%{[@metadata][SONG]}" }

however, the split filter does not support sprintf references in the field option, so that will not work. What you can do is

    grok { match => { "message" => "(?<[@metadata][SONG]>SONG[_A-Z]*)" } }
    json {
        source => "message"
        target => "[@metadata][data]"
        remove_field => [ "message" ]
    }
    ruby {
        code => '
            fieldName = event.get("[@metadata][SONG]")
            data = event.get("[@metadata][data][#{fieldName}]")
            event.set("[@metadata][dataArray]", data)
        '
    }
    split { field => "[@metadata][dataArray]" }
    ruby {
        code => '
            fieldName = event.get("[@metadata][SONG]")
            data = event.get("[@metadata][dataArray]")
            event.set(fieldName, data)
        '
    }
1 Like
3

Thanks Badger, your explanation was clear, especially the following

The code you provided also managed to rectify the problem i was facing.
Thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.