Access dynamic field


In my event, I have a field "end_date_null_count" => 7 and then I use the mutate plugin to add another field "last_null_value" => "column7" :

mutate {
                add_field => {"last_null_value" => "column%{end_date_null_count}"}

My goal is to be able to access to this dynamic field ("last_null_value") in my Grok filter plugin :

 grok {
                match => ["[end_date_null][%{last_null_value}]", "%{TIMESTAMP_ISO8601:timestamp_last} %{GREEDYDATA:definition_last} \[%{LOGLEVEL:log_level_last}\] %{GREEDYDATA:operation_type_last} : %{GREEDYDATA:msg_last}"]

Using this syntax give me a "_grokparsefailure" I have tried another syntax [end_date_null.%{last_null_value}] but but seems that it's not working within the grok filter
end_date_null field :

"end_date_null" => {
    "column2" => "2018-12-13T11:41:44.846+0000 Regulatory [INFO] Transaction : VALIDATE,qf16ft787bif1xs1iuoqihwi9,null,100002506,13-12-2018,13-12-2018T11:41:42.447+0000,null,Payment Order,Date not a working day",
    "column3" => "2018-12-13T12:07:41.644+0000 Regulatory [INFO] Transaction : VALIDATE,007069643021retfed8ar2w4ugvgz1n9xsuz,0070696430.2,100002506,13-12-2018,13-12-2018T12:07:39.905+0000,null,Payment Order,None",
    "column4" => "2018-12-13T13:13:22.449+0000 Regulatory [INFO] Transaction : VALIDATE,0004961017bb48fydx3gvq1dopa7tujzoya,0004961017,100002506,13-12-2018,13-12-2018T13:13:21.700+0000,null,Payment Order,Invalid end date",
    "column5" => "2018-12-13T13:51:13.164+0000 Regulatory [INFO] Transaction : VALIDATE,0004961017121v2xs7x08f975bswzkiv5nona,0004961017.12,100002506,13-12-2018,13-12-2018T13:51:11.773+0000,null,Payment Order,None",
    "column7" => "2018-12-13T13:54:40.123+0000 Regulatory [INFO] Transaction : VALIDATE,007069643021v2xs7x08f975bswzkiv5nona,0070696430.2,100002506,13-12-2018,13-12-2018T13:54:40.469+0000,null,Payment Order,None",
    "column1" => "2018-12-13T11:46:13.654+0000 Regulatory [INFO] Transaction : VALIDATE,FT18260HNC8R1bbcffrlnrt21x8awmxhtjfdz,FT18260HNC8R,100002506,13-12-2018,13-12-2018T11:46:13.243+0000,null,Payment Order,Date not a working day",
    "column6" => "2018-12-13T13:51:17.146+0000 Regulatory [INFO] Transaction : PROCESS,0004961017121v2xs7x08f975bswzkiv5nona,0004961017.12,100002506,13-12-2018,13-12-2018T13:51:16.819+0000,null,Payment Order,None"

Can anyone help me to find the right syntax?
Thank you

I cannot get a sprintf reference to work there, but you could do it using ruby.

    ruby {
         code => '
            wanted = event.get("last_null_value")
            event.get("end_date_null").each { |k, v|
                if k == wanted then
                    event.set("[@metadata][wanted]", v)

You are using an unanchored grok pattern with three GREEDYDATA fields. That's going to be really expensive. You might do better with

    grok {
        match => ["[@metadata][wanted]", "^%{TIMESTAMP_ISO8601:timestamp_last} %{NOTSPACE:definition_last} \[%{LOGLEVEL:log_level_last}\] (?<operation_type_last>[^:]+) : %{GREEDYDATA:msg_last}"]

Thank you @Badger, That works very fine for me.
I changed also the grok pattern.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.