Parse json file with 100k/1 million records?

[Abdul_Gaffar_Shaikh]

hi @Badger,

we have solved the issue , but now i am facing a different issue which is smilar to below link

Can logstash handle array of nested objects?

I am unable to configure logstash to load an array of nested objects as described in the elasticsearch guide. I can map and load the array of nested objects via the elasticsearch REST API (as shown below), but I am unable to do the same via logstash when there is an unknown number of nested objects in the array (in this case, an unknown number of comments). Any suggestions? PUT /my_index/blogpost/1 { "title": "Nest eggs", "body": "Making your money work...", "tags": [ "cash", "shares"…

some fields are added to the end of json like this

{"docId": "201","docOwner": "R909168111","docType":"delay","event1Timestamp":"2019-08-20T04:28:40.889Z", "addinfo":[{"key": "clid","value": "p111"}]
}

filter {

mutate {
strip => ["message"]
}

if [message] == ']}' {
drop{}
}

mutate { gsub => ["message",",$",""] }

json {
source => "message"
}

}

but how to extract the nested array of inner objects and inserted as it is elasticsearch?

i get the output in logstash debug as

"addInfo" => [
[0] {
"key" => "clid",
"value" => "p111"
},
[1] {
"key" => "clid1",
"value" => "p222"
}
],

but in elasticsearch the o/p is :-

"addInfo" : [
{
"key" : "clid",
"value" : "p111"
},
{
"key" : "clid1",
"value" : "p222"
}
],

is that fine ?