# Parse json file with 100k/1 million records?

**URL:** <https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659>\
**Category:** Logstash\
**Created:** [October 29, 2019, 11:58am UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659 "2019-10-29T11:58:32Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [October 29, 2019, 11:58am UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/1 "2019-10-29T11:58:32Z")

</div>

i am trying to parse big json file , here is the sample 5 records in the file with the orignial file containing 100k

{"dgdocs": [{"doc id": "0", "docowner": "0", "documentType": "cte1\_0", "docver": "1", "ogsys": "fc", "docstat1": "smartsys", "claimId": ""}, {"doc id": "1", "docowner": "1", "documentType": "cte1\_0", "docver": "1", "ogsys": "fc", "docstat1": "smartsys", "claimId": ""}, {"doc id": "2", "docowner": "2", "documentType": "cte1\_0", "docver": "1", "ogsys": "fc", "docstat1": "smartsys", "claimId": ""}, {"doc id": "3", "docowner": "3", "documentType": "cte1\_0", "docver": "1", "ogsys": "fc", "docstat1": "smartsys", "claimId": ""}, {"doc id": "4", "docowner": "4", "documentType": "cte1\_0", "docver": "1", "ogsys": "fc", "docstat1": "smartsys", "claimId": ""}], "totalCount": 5}

here is my config script :-

```
 input 
 {  
   file
   {
	
	path => "${LS_XML_PATH}/*.json"
	start_position => "beginning"
	
	sincedb_path => "NUL"
    exclude => "*.gz"
	mode => "read"
	
	#codec => json_lines // tried this with while commenting other codecs
             #code => json // tried this with while commenting other codecs
    codec => multiline // tried this with while commenting other codecs
    {
        pattern => '^\{'
        #pattern => "^}"
     #pattern => "dgdocs"
         negate => true
        what => previous 
        max_lines => 2000000
  auto_flush_interval => 3
    }

}  
 }

  filter
   {
json
  {
    source => "message"
  }

 if [dgdocs]
{
    split 
    {
    id => "cte1main_split_field"
    field => "[dgdocs]"
    }
}

 mutate { gsub => ["message","^(.*)dgdocs: ",""]}

mutate
{
    add_field =>  
    { 
    "creationDate" =>	"%{[@metadata][creationDate]}"
    "docId" =>	"%{[dgdocs][docId]}"
    "docOwner" =>	"%{[dgdocs][docOwner]}"
    "docType" =>	"%{[dgdocs][docType]}"
    "docVersion" =>	"%{[dgdocs][docVer]}" 
    "ogsys" =>	"%{[dgdocs][ogsys]}" 
    "docStatus1" =>	"%{[dgdocs][docStat1]}"
    "event1Timestamp" =>	"%{[@metadata][creationDate]}"
    "claimId" =>	"%{[dgdocs][claimId]}"
    
    }
}

  mutate 
{
     remove_field => 
 ["message","tags","@timestamp","host","@version","path","dgdocs"]

}

 }

  output
 {
#for debugging purpose
stdout	{ codec => rubydebug }

}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 29, 2019, 3:50pm UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/2 "2019-10-29T15:50:05Z")

</div>

Do you have a question?

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [October 29, 2019, 4:09pm UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/3 "2019-10-29T16:09:27Z")

</div>

@Badger,

i have updated my scripted which is working till 1000 records after which when i try with 5000 records and it fails with below error : -

```
buffer_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=>"\n", "read_position"=>1015808, "bytes_read_count"=>12009, "last_known_file_size"=>1027817, 

```

and Also gives OOM ,

```
 java.lang.OutOfMemoryError: Java heap space  
Dumping heap to java_pid12264.hprof ...   
Heap dump file created [2224914329 bytes in 16.233 secs]
warning: thread "[main]>worker0" terminated with exception (report_on_exception is 
true):
java.lang.OutOfMemoryError: Java heap space    
  warning: thread "[main]<file" terminated with exception (report_on_exception is true):
java.lang.OutOfMemoryError: Java heap space 

```

could you please help ?

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [October 31, 2019, 10:39am UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/4 "2019-10-31T10:39:16Z")

</div>

hi @Badger,

we have modified the json in below format :

{"tc":"2","dg":[  
{"docId": "8956","docOwner": "R909168111","docPath": "","docType":"clm"},  
{"docId": "5623","docOwner": "R909168112","docPath": "","docType": "clm"},  
{"docId": "4152","docOwner": "R909168113","docPath": "","docType": "clm"}  
]}

for that i am using below script :\_

input {  
file {  
path =\> /path/to/json  
codec =\> plain  
}  
}

filter {  
json  
{  
source =\> "message"  
}

if [totalCount] {  
ruby {  
code =\> '  
tc= event.get("tc")  
logger.info("the total count:", "tc" =\> tc)  
'  
}  
drop {}  
}

}

output {  
stdout { codec =\> rubydebug }

elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "test-q4-2019"  
}  
}

there will be million records like this and i see error like this

rror parsing json {:source=\>"message", :raw=\>"]}", :exception=\>#\<LogStash::Json::ParserError: Unexpected close marker ']': expected '}' (for root starting at [Source: (byte)"]}"; line: 1, column: 0])

:exception=\>#\<LogStash::Json::ParserError: Unexpected character (',' (code 44)): expected a value  
at [Source: (byte)"

please help

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 31, 2019, 2:33pm UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/5 "2019-10-31T14:33:02Z")

</div>

If your JSON is spread across multiple lines then you will need to use a multiline codec to combine them.

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [November 4, 2019, 10:02am UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/6 "2019-11-04T10:02:43Z")

</div>

hi @Badger,

we have solved the issue , but now i am facing a different issue which is smilar to below link

> [@Can logstash handle array of nested objects?](https://discuss.elastic.co/t/can-logstash-handle-array-of-nested-objects/53558):
>
> I am unable to configure logstash to load an array of nested objects as described in the [elasticsearch guide](https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html). I can map and load the array of nested objects via the elasticsearch REST API (as shown below), but I am unable to do the same via logstash when there is an unknown number of nested objects in the array (in this case, an unknown number of comments). Any suggestions? PUT /my\_index/blogpost/1 { "title": "Nest eggs", "body": "Making your money work...", "tags": [ "cash", "shares"…

some fields are added to the end of json like this

{"docId": "201","docOwner": "R909168111","docType":"delay","event1Timestamp":"2019-08-20T04:28:40.889Z", "**addinfo":[{"key": "clid","value": "p111"}**]  
}

filter {

mutate {  
strip =\> ["message"]  
}

if [message] == ']}' {  
drop{}  
}

mutate { gsub =\> ["message",",$",""] }

json {  
source =\> "message"  
}

}

but how to extract the nested array of inner objects and inserted as it is elasticsearch?

i get the output in logstash debug as

"addInfo" =\> [  
[0] {  
"key" =\> "clid",  
"value" =\> "p111"  
},  
[1] {  
"key" =\> "clid1",  
"value" =\> "p222"  
}  
],

but in elasticsearch the o/p is :-

"addInfo" : [  
{  
"key" : "clid",  
"value" : "p111"  
},  
{  
"key" : "clid1",  
"value" : "p222"  
}  
],

is that fine ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 4, 2019, 1:24pm UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/7 "2019-11-04T13:24:59Z")

</div>

> [@Abdul\_Gaffar\_Shaikh](#):
>
> is that fine ?

Yes, rubydebug and elasticsearch have different ways of displaying arrays.

---

<div class="post-metadata">

**Author:** ![Abdul\_Gaffar\_Shaikh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdul_gaffar_shaikh/32/78006_2.png) [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\
**Post date:** [November 4, 2019, 1:27pm UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/8 "2019-11-04T13:27:50Z")

</div>

Thanks @badger

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 2, 2019, 1:27pm UTC](https://discuss.elastic.co/t/parse-json-file-with-100k-1-million-records/205659/9 "2019-12-02T13:27:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.