JSON nested field loosing formating when referenced in add_field

[marc1]

Hello, we have a use case where we would like to ingest specific JSON formatted messages into elasticsearch as well as convert the content of the input message into a new output JSON format and write it out to a Kafka topic.

The ingestion into elasticsearch as well as writing the output to the Kafka topic is functional. What I am having difficulty with is the conversion from one JSON schema to the other as I have a nested JSON field. I am currently writing out the new JSON message manually in the pipeline by mutating and adding a new field.

Here is an example of the input message:

{
       "sourceSystemId" : "SystemABC", 
       "externalAlarmId" : "11082807", 
        "alarmRaisedTime" : "2018-01-01T19:41:01Z",
        "alarmType" : "QueueStatus",
        "alarmedObject" : {
                  "id": "DeviceABC", 
                  "resource" : { 
                      "id": "DeviceABC", 
                      "name": "DeviceABC",
                      "category": "NODE",
                      "characteristic" : [
                          {
                         "name": "NodeAlias",
                         "value": "192.168.1.1"
                          },
                          {
                         "name": "AlertKey",
                         "value": "c3-1-1-2"
                          },
                          {
                         "name": "deviceType",
                         "value": "router"
                          }
                         ] 
                   }
           },
       "perceivedSeverity" : "MINOR",
}

By using the JSON codec on the input all fields are extracted and usable with the exception of the nested JSON which creates has an array called:

[alarmedObject][resource][characteristic]

When I construct my new JSON message utilizing this array field the output is stripped of the quotes and colon. Here is my sample config as well as actual output and desired output for reference.

input {
        udp {
                tags => ["udp"]
                port => 50000
                codec => json
        }
}
filter {
mutate {
        add_field => {test_message_3 => '{"additionalAlarmInfo":[%{[alarmedObject][resource][characteristic]}]}'}
}

Current OUTPUT when referencing the contents of nested JSON field. We see that for some reason an "=" sign is added to the content.

{
   "additionalAlarmInfo":[
      "{name=NodeAlias, value=192.168.1.1},{name=AlertKey, value=c3-1-1-2},{name=deviceType, value=router}"
   ]
}

Desired OUTPUT which is JSON compliant.

{
   "additionalAlarmInfo":[
      {
  "name": "NodeAlias",
  "value": "192.168.1.1"
},
{
  "name": "AlertKey",
  "value": "c3-1-1-2"
},
{
  "name": "deviceType",
  "value": "router"
}
   ]
}

My question is how do I include the as JSON formatted content of the array into a new field? Am I calling the array incorrectly with the following in my new message?

'{"additionalAlarmInfo":[%{[alarmedObject][resource][characteristic]}]}'

I hope this makes sense. Any direction from the community that could be provided would be greatly appreciated. Thank you.

[magnusbaeck (Magnus Bäck)]

I don't think you can construct what you want using the regular Logstash configuration language. You'll have to use a ruby filter.

[marc1]

Hello, Thank you for the reply and input.

After much trial and error I found the json_encode plugin and was able to use it as desired. By using:

json_encode {
                        source => "[alarmedObject][resource][characteristic]"
                        target => "encoded_additional_information"
 }

The field [alarmedObject][resource][characteristic] is formatted to the following array:

[{"name":"NodeAlias","value":"192.168.135.95"},{"name":"AlertKey","value":"c3-1-1-2"},{"name":"deviceType","value":"AMAS"},{"name":"Count","value":"10058"}]

Which is the proper syntax I am looking for. I then reference the newly encoded field by using %{encoded_additional_information}. The end JSON message with the array looks like the following which is desirable.

"faultFields":{
         "eventSeverity":"MINOR",
         "alarmCondition":"UNACKNOWLEDGED",
         "faultFieldsVersion":"4.0",
         "specificProblem":"EventProcessor :  QueueSize: 0 (Min: 0 Max: 0) DeltaQueue: 0",
         "alarmAdditionalInformation":[
            [
               {
                  "name":"NodeAlias",
                  "value":"192.168.1.1"
               },
               {
                  "name":"AlertKey",
                  "value":"c3-1-1-2"
               },
               {
                  "name":"Count",
                  "value":"10058"
               }
            ]
         ]
      }

Thanks again.