Unable to parse nested JSON using logstash

[Sameer_Panicker]

Nested JSON response -

{
"application": "FinClient",
"occurredOn": "2019-10-07T10:38:39",
"host": "U8HPP213",
"finEvents": [
{
"@xmlns": "http://jabber.org/protocol/pubsub",
"data": {
"apiErrors": {
"apiError": {
"errorData": "78",
"errorMessage": "CF_REQUEST_TIMEOUT_REJECTION",
"errorType": "Device Timeout"
}
}
},
"event": "put",
"requestId": "",
"source": "/finesse/api/Dialog/52031"
},
"Error Handler Invoked for the msgEvent with reason code:78"
],
"userName": "T01232D"
}

Logstash Configuration -
json
{
source => "message"
}

Issue - I am getting a nested JSON response and inside an array. The array can either have single line statements or can have mix of JSON or statements. I would like to extract the errorData value if a JSON entry is returned, else return the statements. The logstash configuration works fine if there are only statements returned. The issue arises if JSON is present. Tried changing all the json configuration under logstash as per other forum queries, but still no luck. At the end after pulling the data, I expect it to come under Kibana as a new field for errorData (78) and the entire json as a string entry under messages.

Can anyone please let me know how to achieve this ?

[Badger]

A json filter has no problem parsing that

   "userName" => "T01232D",
"application" => "FinClient",
  "finEvents" => [
    [0] {
           "source" => "/finesse/api/Dialog/52031",
        "requestId" => "",
            "event" => "put",
           "@xmlns" => "http://jabber.org/protocol/pubsub",
             "data" => {
            "apiErrors" => {
                "apiError" => {
                       "errorData" => "78",
                       "errorType" => "Device Timeout",
                    "errorMessage" => "CF_REQUEST_TIMEOUT_REJECTION"
                }
            }
        }
    },
    [1] "Error Handler Invoked for the msgEvent with reason code:78"
],
       "host" => "U8HPP213",
 "occurredOn" => "2019-10-07T10:38:39"

It sounds like you want to iterate over finEvents, but what exactly do you want to do for the hashes and for the strings?

[Sameer_Panicker]

@Badger

errorData

Thanks for the quick reply.

I also saw that the log is getting parsed successfully through logstash. But the event does'nt shows up in Kibana. I am unable to retrieve any error associated with it, if any.

I am only looking for retrieving the errorData , errorType and errorMessage. Other strings can be dropped or ignored.

[Badger]

@Sameer_Panicker

I am only looking for retrieving the errorData , errorType and errorMessage. Other strings can be dropped or ignored.

You are still not saying what you want. Would this work?

"finEvents" => [
[0] {
"errorData" => "78",
"errorType" => "Device Timeout",
"errorMessage" => "CF_REQUEST_TIMEOUT_REJECTION"
},
[1] "Error Handler Invoked for the msgEvent with reason code:78"
],

if not, then what do you want?

[Sameer_Panicker]

This wont help, because in Kibana how can I see the errorData or type ? I expect an indexed field for errorType and errorData under Kibana so that I can query or create visualization out of it.

I need to extract the json field (errorData) and represent it as an new field.

[Badger]

Please tell me what you want the result to look like. I am not going to keep guessing.

[Sameer_Panicker]

Logstash Response -

"userName" => "T01232D",
"errorData" => "78",
"errorType" => "Device Timeout",
"errorMessage" => "CF_REQUEST_TIMEOUT_REJECTION",
"host" => "U8HPP213",
"occurredOn" => "2019-10-07T10:38:39"

[Sameer_Panicker]

@Badger - Could you please let me know if my response is not clear ?

[Badger]

You could use something like

    ruby {
        code => '
            finEvents = event.get("finEvents")
            if finEvents
                finEvents.each { |x|
                    if x["data"]
                        event.set("errorMessage", x["data"]["apiErrors"]["apiError"]["errorMessage"])
                        event.set("errorData", x["data"]["apiErrors"]["apiError"]["errorData"])
                        event.set("errorType", x["data"]["apiErrors"]["apiError"]["errorType"])
                    end
                }
            end
        '
    }

[Sameer_Panicker]

Thats cool !!! Thanks Badger

I will give it a try and let you know.