# Elasticsearch parallel bulk using Python - issue with json

**URL:** https://discuss.elastic.co/t/elasticsearch-parallel-bulk-using-python-issue-with-json/146026
**Category:** Elasticsearch
**Created:** [August 25, 2018, 3:18pm UTC](https://discuss.elastic.co/t/elasticsearch-parallel-bulk-using-python-issue-with-json/146026 "2018-08-25T15:18:10Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![hari\_priya](https://avatars.discourse-cdn.com/v4/letter/h/f9ae1b/32.png) [@hari\_priya](https://discuss.elastic.co/u/hari_priya)
#### Post date: [August 25, 2018, 3:18pm UTC](https://discuss.elastic.co/t/elasticsearch-parallel-bulk-using-python-issue-with-json/146026/1 "2018-08-25T15:18:10Z")

</div>

Hi All -

I am a newbie with ELasticsearch and I am encountering strange issue .  
Specifications: I have a Json file of size: 0.5 GB  
and I am using python 3.6 , ELasticsearch 6.3 version .  
I am using parallel bulk call .

code:

try:  
deque(helpers.parallel\_bulk(es,read\_json(filename),request\_timeout=60,raise\_on\_error=True,raise\_on\_exception=True), maxlen=0)  
except TransportError as e:  
print(next(read\_json(filename)))

issue#1: I am encountering message saying :  
POST [https://XXXXXXXXXXXXXXXXXX/\_bulk](https://XXXXXXXXXXXXXXXXXX/_bulk) [status:413 request:192.868s]

And encountering an exception and the job is failing /missing inserting some data .

How I can handle this one programatically?  
How can i specify to print out/redirect the records that are getting dropped programatically?

issue#2:  
when I am using another bigger Json file which is of 2 GB size ( which is larger compared to the previous one but exact same format) , it is not throwing any exceptions and inserting everything .

Am i missing something here ? Not sure what is the issue .

Any thoughts for me . I really appreciate all your time and help .

---

<div class="post-metadata">

### Author: ![jaddison](https://avatars.discourse-cdn.com/v4/letter/j/e5b9ba/32.png) [@jaddison](https://discuss.elastic.co/u/jaddison)
#### Post date: [September 10, 2018, 3:50am UTC](https://discuss.elastic.co/t/elasticsearch-parallel-bulk-using-python-issue-with-json/146026/2 "2018-09-10T03:50:31Z")

</div>

So, the format of the two JSON files may be the same, but the content obviously isn't. Elasticsearch is choking on document data from the smaller file because the [data is "too large"](https://github.com/elastic/elasticsearch/blob/237650e9c054149fd08213b38a81a3666c1868e5/server/src/main/java/org/elasticsearch/rest/RestStatus.java#L383) (\<--- status code `413` is `REQUEST_ENTITY_TOO_LARGE`).

To find out which record it is, you might be able to just do a quick look through the file to find the longest line(s), perhaps? (I'm not sure how they're stored, but I assume a doc per line).

Otherwise, you could just avoid the convenience of `parallel_bulk(...)` and code it up yourself, thereby finding which line blows it up.

Or, you could set a debug breakpoint in the Python code where the exception is caught to see the doc.

Or, you might be able to modify the Elasticsearch configuration to allow larger document payloads via REST.

Hope this helps.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 8, 2018, 3:50am UTC](https://discuss.elastic.co/t/elasticsearch-parallel-bulk-using-python-issue-with-json/146026/3 "2018-10-08T03:50:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.