Add Nested Field Value to a new field using Logstash Filter

Hello All,

I hope someone here could help me as I have been stuck for days on this issue I am facing. I have got the kibana output as shown below:

I am trying to add the nested field value of "key" into a new field called key. This is what I am trying to do using logstash filter:

filter {
   mutate {
       add_field => {
          "key"   => "%{[properties][additionalDetails][key]}"
      }
   }
}

However the below is giving me the output:

Any ideas, where I am going wrong with this one? Any help would be much appreciated. Thanks.

It is inconceivable to me that

add_field => { "key"   => "%{[properties][additionalDetails][key]}" }

would result in key being set to

"%{[properties.additionalDetails][key]}"

logstash is not going to replace the ][ with .

Most likely that mutate will work as is.

Hi Badger,

Thank you for coming back to me. Much appreciated

Sorry for the confusion. The last screenshot is incorrect. The actual output in Kibana is showing:

key             "%{[properties][additionalDetails][key]}" 

instead of:

key              User-Agent

I have edited my original post with the correct screenshot (error)

Can you run logstash with

output { stdout { codec => rubydebug } }

and show us that the [properties] field looks like?

Hi Badger,

Seems like I am not getting any console output with the debugging.

I have even done:

output {
  stdout {
      path => /var/log/logstash/tmp/ruby_output
      codec => rubydebug
 }
}

Still nothing in the file and looking in the logstash-plain.log, nothing related to stdout is in there. The last log just states "successfully started logstash API endpoint"

Even ran logstash as:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/output.conf

and no joy with stdout

i can show you the json output of what kibana is showing me instead if that helps?

The stdout output does not have a path output. Use a file output if you want to specify a path.

The JSON from kibana does tell me enough. Note that additionalDetails is an array. Try

add_field => { "key"   => "%{[properties][additionalDetails][0][key]}" }
1 Like

Hi Badger,

Ahhh perfect. That worked! Thank you.

I should have tried that before but instead I put it as:

add_field => { "key"   => "%{[properties][additionalDetails][key][0]}" }