How to parse stringified json in logstash

Elastic Stack Logstash
1 
{
    "name": "cmg-notification-service App",
    "hostname": "notify-service-main-8cbfffc56-dq924",
    "pid": 1,
    "crn": "XXXXXXXXXXX",
    "url": "/v2/notify/sms",
    "requestId": "IJaxVCtAilpjSRODkySgaQ==",
    "level": 30,
    "msg": "Kafka payload for PersonsNotificationIssued {\n  metadata: {\n    version_number: '1.0.0',\n    lob_schema: '2012',\n    host_name: 'notifyservice.XXX.XXX-XXX.XXX-eks.XXXcloud.uk',\n    ip_address: 'notifyservice.cms.XXX-dev.XXX-eks.XXXcloud.uk',\n    sor: 'CMG Notification Service',\n    uri: 'https://notifyservice.cms.XXX-dev.XXX-eks.XXXcloud.uk/v2/notify',\n    transaction_parent_id: 'c4cd7290-6452-4cfe-9531-da79b190d8cd',\n    transaction_child_id: '756XXXXXXX',\n    transaction_date: '2024-09-27T08:24:36.312Z'\n  },\n  business_interest: {\n    customer_ref_number: '{\"DEK\":\"NcGRIjKZPckaf5f+trwO5s/3uDTIB6YC6dfIklFXbWr/e+4OBJAv/HEIdI2L++o6xmvG3Z0KD8Q8bm/iiwIyRw==\",\"message\":\"NcGRIjKZPckaf5f+trwO5mqo5BH1Xs8UeAv/7OttQK7hX5/YnAlT/n/mYkQ=\",\"crypto_metadata\":{\"group\":\"CentralGovRef\",\"version\":\"1\"}}',\n    transaction_ref_number: '{\"DEK\":\"RskioQAlkcZXC3burgPxHlahWpz7ip7pmLcW1h5QugTBKcGH4EDeLObrYidR25KhoSSj4Df69xLASlU4BekcPg==\",\"message\":\"RskioQAlkcZXC3burgPxHjtfjp/izwKpsspu6e0WXAJ74O6pgJpdeNCrmwY1\",\"crypto_metadata\":{\"group\":\"CentralGovRef\",\"version\":\"1\"}}',\n    notification_ref_number: 'c4cd7290-6452-4cfe-9531-da79b190d8cd',\n    notification_channel: 'SMS',\n    templateId: 'S0001',\n    templateVersion: 1,\n    emailNotificationPreferred: true,\n    source: 'XXXXX XXXXX XXXXX',\n    failure_reason: undefined,\n    AdditionalInformation: { transactionRef: '7XXXXXXXX', amount: '5.45' }\n  }\n}",
    "time": "2024-09-27T08:24:36.314Z",
    "v": 0
}


This is a log (message)i have in cloud watch and i want to parse using json.
when i used "json {source => message} " it parses some which are in JSON format except "msg".
when i use json {source => msg} to further parse it throws me an error _jsonparsefailure

I came to know that this is in stringified json, how to parse this ?
2

No, it is not stringified JSON, although it does have stringified JSON embedded in it (the two ref_number fields). You can try parsing it with a kv filter

    json { source => "message" remove_field => [ "message" ] }

    kv {
        source => "msg"
        field_split_pattern => "\\n" # Or perhaps "\n"
        value_split_pattern => ": "
        trim_key => " "
        trim_value => ",'"
        remove_field => [ "msg" ]
    }
    json { source => "transaction_ref_number" target => "transaction_ref_number" }
    json { source => "customer_ref_number" target => "customer_ref_number" }

Note that the structure of fields within the business_interest and metadata objects is lost. Everything gets flattened. If that is a problem then another approach would be

    json { source => "message" remove_field => [ "message" ] }

    grok {
        break_on_match => false
        match => {
            msg => [
                "metadata: %{GREEDYDATA:[@metadata][metadata]}},",
                "business_interest: %{GREEDYDATA:[@metadata][bi]}\n  }"
            ]
        }
        #remove_field => [ "msg" ]
    }
    kv {
        source => "[@metadata][metadata]"
        target => "metadata"
        field_split_pattern => "\\n"
        value_split_pattern => ": "
        trim_key => " "
        trim_value => ",'"
    }
    kv {
        source => "[@metadata][bi]"
        target => "business_interest"
        field_split_pattern => "\\n"
        value_split_pattern => ": "
        trim_key => " "
        trim_value => ",'"
    }
    json { source => "[business_interest][transaction_ref_number]" target => "[business_interest][transaction_ref_number]" }
    json { source => "[business_interest][customer_ref_number]" target => "[business_interest][customer_ref_number]" }

but those grok patterns will be fragile against changes in the message format.

3

Thanks @Badger I will try the above, and by the way what is kv filter? (I am new to ELK)

4

It parses key/value pairs. The documentation is here.