Nested json with multi field parsing through logstash

sudhir_singh · October 23, 2023, 8:50am

sudhir_singh · October 23, 2023, 8:52am

I want a output like below SS.

and I want the output like this below formatted json:

leandrojmp · October 23, 2023, 1:39pm

What have you tried? Not sure what is the issue, this is a json message you need to use the json filter to parse it.

sudhir_singh · October 26, 2023, 5:58am

It's not working, It's not parsing it properly it's throwing _jsonparsefailure

leandrojmp · October 26, 2023, 12:27pm

But what does your configuration looks like? You didn't share anything about the Logstash configuration you are using.

You need to provide some context, what is your input? What is the configuration you are using? Is your json one document per line or it is pretty printed in your input?

Please share your Logstash configuration.

leandrojmp · October 26, 2023, 12:52pm

I had no issue parsing this json.

[2023-10-26T09:51:31,704][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-10-26T09:51:31,756][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
             "ecs" => {
        "version" => "1.6.0"
    },
        "sitename" => "nbt_app",
      "@timestamp" => 2023-10-11T07:38:56.607Z,
            "host" => "lab",
       "log.level" => "error",
            "zone" => "vsp1",
          "caller" => "res.send:122",
    "requestedAPI" => [
        [0] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/74932839?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1 POTIME=1.677sec",
        [1] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546550?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1 POTIME=1.666sec",
        [2] "http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104228058&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow POTIME=1.983sec",
        [3] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/3531240?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1 POTIME=2.086sec",
        [4] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/2279790?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1 POTIME=2.082sec",
        [5] "https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1 POTIME=2.091sec",
        [6] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546550?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1 POTIME=2.059sec",
        [7] "https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=sports&days=3&perpage=6&mode=medium POTIME=2.065sec",
        [8] "https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1 POTIME=2.092sec",
        [9] "https://cloudservices.indiatimes.com/cms-api/detail/104045553,104226596,104050938?hostid=53 POTIME=0.213sec"
    ],
          "requrl" => "/app/endofarticle.cms?secmsid=2279790&msid=104228058&property=NBT&platform=android&vrsn=4600&islive=yes&adsec=sports",
             "env" => "production",
        "@version" => "1"
}
[2023-10-26T09:51:32,013][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}

sudhir_singh · October 31, 2023, 10:54am

leandrojmp · October 31, 2023, 12:05pm

You didn't, both logs you shared in your previous post have the same format, only this new log is a little different, you also didn't share any error logs, so it is not clear if you are having issues with Logstash or with Elasticsearch.

The following filters will parse all your json messages and split on the field requestedAPI.

filter {
    json {
        source => "message"
    }
    split {
        field => ["requestedAPI"]
    }
    if [requestedAPI][apiUrl] {
        mutate {
            rename => {
                "[requestedAPI]" => "[requestedAPIObject]"
            }
        }
    }
}

The conditional on the previous filter is required because you have conflicting documents, in some of them you have requestedAPI as a string and in others you may have requestedAPI as an object, this is not allowed, will give you mapping errors while trying to index the data in Elasticsearch and the conflicting documents will be rejected, so you need to rename the field when it is an json object.

Another option is to fix this in the source of your data, but this is out of the scope of the forum.

sudhir_singh · November 1, 2023, 11:06am

I'm still getting this error

I used your configuration which is below:

sudhir_singh · November 1, 2023, 11:10am

My json logs are below:

{"@timestamp":"2023-11-01T11:07:52.977Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","apiUrl0":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/74932792?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC0":0.071,"apiUrl1":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/64985397?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC1":0.065,"apiUrl2":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC2":0.078,"apiUrl3":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546453?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC3":0.071,"apiUrl4":"https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1","apiUrlPOTIME_inSEC4":0.066,"apiUrl5":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546453?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC5":0.063,"apiUrl6":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/64679468?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1","apiUrlPOTIME_inSEC6":0.064,"apiUrl7":"https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=travel&days=3&perpage=6&mode=medium","apiUrlPOTIME_inSEC7":0.063,"apiUrl8":"http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104881244&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow","apiUrlPOTIME_inSEC8":0.26,"apiUrl9":"https://cloudservices.indiatimes.com/cms-api/detail/104571048,104855783,104721160?hostid=53","apiUrlPOTIME_inSEC9":0.032,"totalProcesstime_inSec":0.554,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=64679468&msid=104881244&property=NBT&platform=ios&vrsn=6220&islive=yes&adsec=lifestyle"}
{"@timestamp":"2023-11-01T11:07:53.041Z","log.level":"error","message":"Logged in redis file","ecs.version":"1.6.0","tempRequestURI":"/app/endofarticle.cms?secmsid=64679468&msid=104881244&property=NBT&platform=ios&vrsn=6220&islive=yes&adsec=lifestyle POTIME=0.618sec","caller":"res.send:154","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=64679468&msid=104881244&property=NBT&platform=ios&vrsn=6220&islive=yes&adsec=lifestyle"}
{"@timestamp":"2023-11-01T11:07:53.077Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","apiUrl0":"https://cloudservices.indiatimes.com/cms-api/specialsectionhierarchy/69826737?&hostid=53&perpage=30","apiUrlPOTIME_inSEC0":0.016,"apiUrl1":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/69826737?mode=large&hostid=53&contenttype=all&perpage=20&pageno=1","apiUrlPOTIME_inSEC1":0.014,"totalProcesstime_inSec":0.034,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/api_articlelist?msid=69826737&adsec=entertainment&vrsn=4600&platform=android"}
{"@timestamp":"2023-11-01T11:07:53.078Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","apiUrl0":"https://cloudservices.indiatimes.com/cms-api/specialsectionhierarchy/69826737?&hostid=53&perpage=30","apiUrlPOTIME_inSEC0":0.016,"apiUrl1":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/69826737?mode=large&hostid=53&contenttype=all&perpage=20&pageno=1","apiUrlPOTIME_inSEC1":0.014,"totalProcesstime_inSec":0.035,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/api_articlelist?msid=69826737&adsec=entertainment&vrsn=4600&platform=android"}
{"@timestamp":"2023-11-01T11:07:53.245Z","log.level":"error","message":"Got data from redis","ecs.version":"1.6.0","tempRequestURI":"/app/endofarticle.cms?secmsid=33503484&msid=104888115&property=NBT&platform=android&vrsn=4590&islive=yes&adsec=news POTIME=0.003sec","caller":"Command.callback:88","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=33503484&msid=104888115&property=NBT&platform=android&vrsn=4590&islive=yes&adsec=news"}
{"@timestamp":"2023-10-20T10:39:21.825Z","log.level":"error","message":"API REQUEST TIME","ecs.version":"1.6.0","requestedAPI":[{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/58166012?mode=large&hostid=53&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC":0.056},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/75432298?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.065},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/88049765?mode=large&hostid=53&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.052},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/priority/68668311?mode=large&hostid=53&contenttype=all&perpage=2&pageno=1","apiUrlPOTIME_inSEC":0.079},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/33503484?mode=large&hostid=53&contenttype=all&perpage=7&pageno=1","apiUrlPOTIME_inSEC":0.077},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546524?mode=large&hostid=374&contenttype=all&perpage=1&pageno=1","apiUrlPOTIME_inSEC":0.081},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/leafhierarchy/96546524?mode=large&hostid=374&contenttype=all&perpage=5&pageno=1","apiUrlPOTIME_inSEC":0.093},{"apiUrl":"https://cloudservices.indiatimes.com/cms-analytics/mostread?hostid=53&sections=metro&days=3&perpage=6&mode=medium","apiUrlPOTIME_inSEC":0.075},{"apiUrl":"http://miscjcmssolr.indiatimes.com/TOISearchWs/recommendation?wt=json&msid=104581382&hostid=53&fl=title,msid,hostid,contenttypeid,contentsubtypeid,effectivedate,seopath&perpage=3&type=article,video,slideshow","apiUrlPOTIME_inSEC":0.113},{"apiUrl":"https://cloudservices.indiatimes.com/cms-api/detail/104447157,104228861,104531812?hostid=53","apiUrlPOTIME_inSEC":0.042}],"totalProcesstime_inSec":0.353,"caller":"res.send:126","sitename":"nbt_app","zone":"vsp1","env":"production","requrl":"/app/endofarticle.cms?secmsid=33503484&msid=104581382&property=NBT&platform=android&vrsn=4590&islive=yes&adsec=news"}

leandrojmp · November 1, 2023, 12:21pm

This is expected since the messages are different, if you are parsing different messages formats you need to have conditionals.

Check the warning, the message is clear, it says that the requestedAPI field is of the type NilClass, which means that the field does not exist in your document.

If you have a split filter on a field and this field does not exist in your document, you will get this error, this is unrelated and have no impact on the documents that have the field.

You can ignore this warning, if you want to avoid it you can put the split plugin inside a conditional.

    if [requestedAPI] {
        split {
            field => ["requestedAPI"]
        }
    }

sudhir_singh · November 5, 2023, 8:58pm

Thank You so much @leandrojmp for the help.

sudhir_singh · November 5, 2023, 8:59pm

@leandrojmp Can you please help me with below error. I'm facing while intallling fleet and the agent.

Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority

leandrojmp · November 5, 2023, 9:40pm

I suggest that you open a different topic as this is unrelated to your original question.

Also, check this documentation.

sudhir_singh · November 6, 2023, 4:45am

I understand I've already opened the topic for that check the below url:

sudhir_singh · November 10, 2023, 1:31pm

@leandrojmp can please help with this Not able to parse completely with grok - #2 by sudhir_singh

system · December 8, 2023, 1:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.