Add Nested Field Value to a new field using Logstash Filter

[Kevin_f]

Hello All,

I hope someone here could help me as I have been stuck for days on this issue I am facing. I have got the kibana output as shown below:

I am trying to add the nested field value of "key" into a new field called key. This is what I am trying to do using logstash filter:

filter {
   mutate {
       add_field => {
          "key"   => "%{[properties][additionalDetails][key]}"
      }
   }
}

However the below is giving me the output:

Any ideas, where I am going wrong with this one? Any help would be much appreciated. Thanks.

[Badger]

It is inconceivable to me that

add_field => { "key"   => "%{[properties][additionalDetails][key]}" }

would result in key being set to

"%{[properties.additionalDetails][key]}"

logstash is not going to replace the ][ with .

Most likely that mutate will work as is.

[Kevin_f]

Hi Badger,

Thank you for coming back to me. Much appreciated

Sorry for the confusion. The last screenshot is incorrect. The actual output in Kibana is showing:

key             "%{[properties][additionalDetails][key]}" 

instead of:

key              User-Agent

I have edited my original post with the correct screenshot (error)

[Badger]

Can you run logstash with

output { stdout { codec => rubydebug } }

and show us that the [properties] field looks like?

[Kevin_f]

Hi Badger,

Seems like I am not getting any console output with the debugging.

I have even done:

output {
  stdout {
      path => /var/log/logstash/tmp/ruby_output
      codec => rubydebug
 }
}

Still nothing in the file and looking in the logstash-plain.log, nothing related to stdout is in there. The last log just states "successfully started logstash API endpoint"

Even ran logstash as:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/output.conf

and no joy with stdout

i can show you the json output of what kibana is showing me instead if that helps?

[Badger]

The stdout output does not have a path output. Use a file output if you want to specify a path.

The JSON from kibana does tell me enough. Note that additionalDetails is an array. Try

add_field => { "key"   => "%{[properties][additionalDetails][0][key]}" }

[Kevin_f]

Hi Badger,

Ahhh perfect. That worked! Thank you.

I should have tried that before but instead I put it as:

add_field => { "key"   => "%{[properties][additionalDetails][key][0]}" }