# Import large number of json documents failing via bulk import with no error message

**URL:** https://discuss.elastic.co/t/import-large-number-of-json-documents-failing-via-bulk-import-with-no-error-message/154300
**Category:** Elasticsearch
**Created:** [October 27, 2018, 6:43pm UTC](https://discuss.elastic.co/t/import-large-number-of-json-documents-failing-via-bulk-import-with-no-error-message/154300 "2018-10-27T18:43:01Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![BinaryIsPrimary](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/binaryisprimary/32/46210_2.png) [@BinaryIsPrimary](https://discuss.elastic.co/u/BinaryIsPrimary)
#### Post date: [October 27, 2018, 6:43pm UTC](https://discuss.elastic.co/t/import-large-number-of-json-documents-failing-via-bulk-import-with-no-error-message/154300/1 "2018-10-27T18:43:01Z")

</div>

I am trying to import documents stored in a json file (1.4GB) via the bulk api with the following command:

```
 curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/{index}/_bulk --data-binary "@<filename>.json"

```

With small json documents (\< 100MB) it works as expected, but with larger files such as the one mentioned above, it just exits with no message at all. No info in /var/log/elasticsearch/cluster.log at all.

My current cluster (mesh VPN) specs:

Master eligible node: 32GB RAM, 1TB HDD, 4 cores (jvm: Xms16g, Xmx16g)  
Data node 1: 32GB RAM, 1TB HDD, 4 cores (jvm: Xms16g, Xmx16g)  
Data node 2: 32GB RAM, 1TB HDD, 4 cores (jvm: Xms16g, Xmx16g)

Structure of the json documents are:

"  
{"index": {"\_index": "\<index-title\>", "\_type": "\<type-title\>, "\_id": "\<document-id\>"}}  
{"attr01": val01, "attr02: "attr02", [...]}  
\<repeating for all documents\>

"

What could be the issue?

Where can I find log information about the failing import?

---

<div class="post-metadata">

### Author: ![nugusbayevkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nugusbayevkk/32/126683_2.png) [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)
#### Post date: [October 27, 2018, 8:36pm UTC](https://discuss.elastic.co/t/import-large-number-of-json-documents-failing-via-bulk-import-with-no-error-message/154300/2 "2018-10-27T20:36:15Z")

</div>

hi, @BinaryIsPrimary  
you should increase parameter: `"http.max_content_length"` in elasticsearch.yml and restart elasticsearch to needed value, for example

> http.max\_content\_length: "500mb"

by default, it equal to 100mb.  
here [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html)

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 28, 2018, 7:05am UTC](https://discuss.elastic.co/t/import-large-number-of-json-documents-failing-via-bulk-import-with-no-error-message/154300/3 "2018-10-28T07:05:42Z")

</div>

The bulk index API is designed to efficiently index groups of documents in a single request. It is however not designed to necessarily index all documents in one request, and it is generally recommend to keep the size of each bulk request around 5MB or so. I would therefore recommend breaking your file up into multiple smaller ones rather than try increasing the `http.max_content_length` parameter, which can put a lot of load on the cluster.

---

<div class="post-metadata">

### Author: ![BinaryIsPrimary](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/binaryisprimary/32/46210_2.png) [@BinaryIsPrimary](https://discuss.elastic.co/u/BinaryIsPrimary)
#### Post date: [October 28, 2018, 7:17am UTC](https://discuss.elastic.co/t/import-large-number-of-json-documents-failing-via-bulk-import-with-no-error-message/154300/4 "2018-10-28T07:17:21Z")

</div>

That makes sense, I will split them up then. Thank you.

@nugusbayevkk Thanks for your advice, too. I increased the value, but the result is still the same. I will follow @Christian_Dahlqvist's advice.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 25, 2018, 7:31am UTC](https://discuss.elastic.co/t/import-large-number-of-json-documents-failing-via-bulk-import-with-no-error-message/154300/5 "2018-11-25T07:31:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.