Nested json with multi field parsing through logstash

Please help me with below log how do I parse it ?
{"@timestamp":"2023-10-11T07:38:56.607Z","log.level":"error","message":"API REQUEST TIME","ecs":{"version":"1.6.0"},"requestedAPI":["https://cloudservices.indiatimes.com/cms-api/leafhierarchy/74932839?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1 POTIME=1.677sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546550?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1 POTIME=1.666sec","http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104228058&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow POTIME=1.983sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/3531240?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1 POTIME=2.086sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/2279790?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1 POTIME=2.082sec","https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1 POTIME=2.091sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546550?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1 POTIME=2.059sec","https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=sports&days=3&perpage=6&mode=medium POTIME=2.065sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1 POTIME=2.092sec","https://cloudservices.indiatimes.com/cms-api/detail/104045553,104226596,104050938?hostid=53 POTIME=0.213sec"],"caller":"res.send:122","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=2279790&msid=104228058&property=NBT&platform=android&vrsn=4600&islive=yes&adsec=sports"}

I want a output like below SS.

This is another case I have a log like this
{"@timestamp":"2023-10-11T07:38:56.607Z","log.level":"error","message":"API REQUEST TIME","ecs":{"version":"1.6.0"},"requestedAPI":["https://cloudservices.indiatimes.com/cms-api/leafhierarchy/74932839?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1 POTIME=1.677sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546550?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1 POTIME=1.666sec","http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104228058&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow POTIME=1.983sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/3531240?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1 POTIME=2.086sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/2279790?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1 POTIME=2.082sec","https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1 POTIME=2.091sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546550?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1 POTIME=2.059sec","https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=sports&days=3&perpage=6&mode=medium POTIME=2.065sec","https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1 POTIME=2.092sec","https://cloudservices.indiatimes.com/cms-api/detail/104045553,104226596,104050938?hostid=53 POTIME=0.213sec"],"caller":"res.send:122","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=2279790&msid=104228058&property=NBT&platform=android&vrsn=4600&islive=yes&adsec=sports"}

and I want the output like this below formatted json:

What have you tried? Not sure what is the issue, this is a json message you need to use the json filter to parse it.

It's not working, It's not parsing it properly it's throwing _jsonparsefailure

But what does your configuration looks like? You didn't share anything about the Logstash configuration you are using.

You need to provide some context, what is your input? What is the configuration you are using? Is your json one document per line or it is pretty printed in your input?

Please share your Logstash configuration.

I had no issue parsing this json.

[2023-10-26T09:51:31,704][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-10-26T09:51:31,756][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
             "ecs" => {
        "version" => "1.6.0"
    },
        "sitename" => "nbt_app",
      "@timestamp" => 2023-10-11T07:38:56.607Z,
            "host" => "lab",
       "log.level" => "error",
            "zone" => "vsp1",
          "caller" => "res.send:122",
    "requestedAPI" => [
        [0] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/74932839?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1 POTIME=1.677sec",
        [1] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546550?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1 POTIME=1.666sec",
        [2] "http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104228058&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow POTIME=1.983sec",
        [3] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/3531240?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1 POTIME=2.086sec",
        [4] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/2279790?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1 POTIME=2.082sec",
        [5] "https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1 POTIME=2.091sec",
        [6] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546550?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1 POTIME=2.059sec",
        [7] "https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=sports&days=3&perpage=6&mode=medium POTIME=2.065sec",
        [8] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1 POTIME=2.092sec",
        [9] "https://cloudservices.indiatimes.com/cms-api/detail/104045553,104226596,104050938?hostid=53 POTIME=0.213sec"
    ],
          "requrl" => "/app/endofarticle.cms?secmsid=2279790&msid=104228058&property=NBT&platform=android&vrsn=4600&islive=yes&adsec=sports",
             "env" => "production",
        "@version" => "1"
}
[2023-10-26T09:51:32,013][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}

{"@timestamp":"2023-10-20T10:39:21.825Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","requestedAPI":[{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC":0.056},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/75432298?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.065},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/88049765?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.052},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1","apiUrlPOTIME_inSEC":0.079},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/33503484?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1","apiUrlPOTIME_inSEC":0.077},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546524?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC":0.081},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546524?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.093},{"apiUrl":"https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=metro&days=3&perpage=6&mode=medium","apiUrlPOTIME_inSEC":0.075},{"apiUrl":"http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104581382&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow","apiUrlPOTIME_inSEC":0.113},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/detail/104447157,104228861,104531812?hostid=53","apiUrlPOTIME_inSEC":0.042}],"totalProcesstime_inSec":0.353,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=33503484&msid=104581382&property=NBT&platform=android&vrsn=4590&islive=yes&adsec=news"}
That's the problem I have shared two logs format requestedAPI [0] should be a field type "apiURL" and POTIME should be splited and should be a field too. And please share your logstash configuration.

You didn't, both logs you shared in your previous post have the same format, only this new log is a little different, you also didn't share any error logs, so it is not clear if you are having issues with Logstash or with Elasticsearch.

The following filters will parse all your json messages and split on the field requestedAPI.

filter {
    json {
        source => "message"
    }
    split {
        field => ["requestedAPI"]
    }
    if [requestedAPI][apiUrl] {
        mutate {
            rename => {
                "[requestedAPI]" => "[requestedAPIObject]"
            }
        }
    }
}

The conditional on the previous filter is required because you have conflicting documents, in some of them you have requestedAPI as a string and in others you may have requestedAPI as an object, this is not allowed, will give you mapping errors while trying to index the data in Elasticsearch and the conflicting documents will be rejected, so you need to rename the field when it is an json object.

Another option is to fix this in the source of your data, but this is out of the scope of the forum.

I'm still getting this error

I used your configuration which is below:

My json logs are below:

{"@timestamp":"2023-11-01T11:07:52.977Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","apiUrl0":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/74932792?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC0":0.071,"apiUrl1":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/64985397?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC1":0.065,"apiUrl2":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC2":0.078,"apiUrl3":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546453?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC3":0.071,"apiUrl4":"https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1","apiUrlPOTIME_inSEC4":0.066,"apiUrl5":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546453?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC5":0.063,"apiUrl6":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/64679468?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1","apiUrlPOTIME_inSEC6":0.064,"apiUrl7":"https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=travel&days=3&perpage=6&mode=medium","apiUrlPOTIME_inSEC7":0.063,"apiUrl8":"http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104881244&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow","apiUrlPOTIME_inSEC8":0.26,"apiUrl9":"https://cloudservices.indiatimes.com/cms-api/detail/104571048,104855783,104721160?hostid=53","apiUrlPOTIME_inSEC9":0.032,"totalProcesstime_inSec":0.554,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=64679468&msid=104881244&property=NBT&platform=ios&vrsn=6220&islive=yes&adsec=lifestyle"}
{"@timestamp":"2023-11-01T11:07:53.041Z","log.level":"error","message":"Logged in redis file","ecs.version":"1.6.0","tempRequestURI":"/app/endofarticle.cms?secmsid=64679468&msid=104881244&property=NBT&platform=ios&vrsn=6220&islive=yes&adsec=lifestyle POTIME=0.618sec","caller":"res.send:154","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=64679468&msid=104881244&property=NBT&platform=ios&vrsn=6220&islive=yes&adsec=lifestyle"}
{"@timestamp":"2023-11-01T11:07:53.077Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","apiUrl0":"https://cloudservices.indiatimes.com/cms-api/specialsectionhierarchy/69826737?&hostid=53&perpage=30","apiUrlPOTIME_inSEC0":0.016,"apiUrl1":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/69826737?mode=large&hostid=53&contenttype=all&perpage=20&pageno=1","apiUrlPOTIME_inSEC1":0.014,"totalProcesstime_inSec":0.034,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/api_articlelist?msid=69826737&adsec=entertainment&vrsn=4600&platform=android"}
{"@timestamp":"2023-11-01T11:07:53.078Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","apiUrl0":"https://cloudservices.indiatimes.com/cms-api/specialsectionhierarchy/69826737?&hostid=53&perpage=30","apiUrlPOTIME_inSEC0":0.016,"apiUrl1":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/69826737?mode=large&hostid=53&contenttype=all&perpage=20&pageno=1","apiUrlPOTIME_inSEC1":0.014,"totalProcesstime_inSec":0.035,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/api_articlelist?msid=69826737&adsec=entertainment&vrsn=4600&platform=android"}
{"@timestamp":"2023-11-01T11:07:53.245Z","log.level":"error","message":"Got data from redis","ecs.version":"1.6.0","tempRequestURI":"/app/endofarticle.cms?secmsid=33503484&msid=104888115&property=NBT&platform=android&vrsn=4590&islive=yes&adsec=news POTIME=0.003sec","caller":"Command.callback:88","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=33503484&msid=104888115&property=NBT&platform=android&vrsn=4590&islive=yes&adsec=news"}
{"@timestamp":"2023-10-20T10:39:21.825Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","requestedAPI":[{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC":0.056},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/75432298?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.065},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/88049765?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.052},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1","apiUrlPOTIME_inSEC":0.079},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/33503484?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1","apiUrlPOTIME_inSEC":0.077},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546524?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC":0.081},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546524?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.093},{"apiUrl":"https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=metro&days=3&perpage=6&mode=medium","apiUrlPOTIME_inSEC":0.075},{"apiUrl":"http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104581382&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow","apiUrlPOTIME_inSEC":0.113},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/detail/104447157,104228861,104531812?hostid=53","apiUrlPOTIME_inSEC":0.042}],"totalProcesstime_inSec":0.353,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=33503484&msid=104581382&property=NBT&platform=android&vrsn=4590&islive=yes&adsec=news"}

This is expected since the messages are different, if you are parsing different messages formats you need to have conditionals.

Check the warning, the message is clear, it says that the requestedAPI field is of the type NilClass, which means that the field does not exist in your document.

If you have a split filter on a field and this field does not exist in your document, you will get this error, this is unrelated and have no impact on the documents that have the field.

You can ignore this warning, if you want to avoid it you can put the split plugin inside a conditional.

    if [requestedAPI] {
        split {
            field => ["requestedAPI"]
        }
    }

Thank You so much @leandrojmp for the help.

@leandrojmp Can you please help me with below error. I'm facing while intallling fleet and the agent.

Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority

I suggest that you open a different topic as this is unrelated to your original question.

Also, check this documentation.

I understand I've already opened the topic for that check the below url:

@leandrojmp can please help with this Not able to parse completely with grok - #2 by sudhir_singh