Extract nested json

[Chris_Clifton]

I have the following nested json that I'm trying to pull out into top level fields so I can use mutate to add and modify,

These events are picked from an awscloudwatch stream with filebeat v7.10.2 then fed to a logstash v7.10.2 instance on an aws linux ec2 instance,

    "id": "3",
    "type": "TaskScheduled",
    "details": {
        "parameters": "{\"FunctionName\":\"Converters-metrics-prod\",\"Payload\":{\"Type\":\"start\",\"Input\":{\"Bucket\":\"bucket-name\",\"Key\":\"FOLDER/filename.txt\"},\"Id\":\"arn:aws:states:us-west-2:123456789012:execution:converter-prod:81234557-abcd-1234-abcd-1234567abc12\"}}",
        "region": "us-west-2",
        "resource": "invoke",
        "resourceType": "lambda"
    },
    "previous_event_id": "2",
    "event_timestamp": "1613082339534",
    "execution_arn": "arn:aws:states:us-west-2:123456789012:execution:converter-prod:81234557-abcd-1234-abcd-1234567abc12"
}

The bucket name and folder/file name is what I need to pull out into top level fields,

I've tried using the following filter syntax in logstash ,

json {
      source => "message"
    }

    mutate {
      rename => { "type" => "lamda_event" }
    }

    if [lamda_event] == "TaskScheduled" {

         ruby {
          code => "
           begin
            detailsField= event.get('details')
            if detailsField!= NIL
             detailsField.keys.each{|key|
             event.set(key, detailsField[key])
              }
           event.remove('detailsField')
           end
          end
          " }

The ruby code is putting the details.parameters in another field, but not separating the subfields out into top level fields :

field : details.parameters

{"FunctionName":"Converters-metrics-prod","Payload":{"Input":
{"Bucket":"bucket-name","Key":"FOLDER/filename.txt"},
"Type":"finished","Id":
"arn:aws:states:us-west-2:123456789012:execution:converter-prod:81234557-abcd-1234-abcd-1234567abc12"}}

I've also tried using the following , but I can't seem to get the correct syntax to pull out the sub fields in the nested json

commented for now since I was trying the ruby code above,

filter {
             json {
               source => "details.parameters"
               target => "TaskScheduledParameters"
              }
              mutate {
                add_field => {
                    "FunctionName" => "%{[TaskScheduledParameters][FunctionName]}"
                }
          }
        }

Thanks!

[Chris_Clifton]

Answering my own question here, and hoping this might help someone in the future, the ruby code I was using , not necessary at all.

If your json is a little rusty, you may (as I was) be used to 'regular' text fields in logstash being referred to as ["field"].

Importantly, in logstash, regular fields are referenced ["field"] , once you apply the logstash json filter, the json fields are referred to using "[jsonfield]". Note the placement of brackets and quotes. Then as you go down the nesting it's "[json][nestedfield]". So that's what I was missing, trying to refer to json fields with the wrong syntax. Once I started using the "[json][nestedfield]", everything started working as expected.

Once you've applied the json filter, the fields are broken out mostly as expected, and you may see the fields referenced as field.subfield in kibana. Clicking on the 'json' tab in kibana will help you see how these are actually being referred to, they are subfields, not just a field named field.subfield.