# \[elastic/elasticsearch\] Cannot bulk index a JSON file greater than 100MB in Elasticsearch. Tried changing HTTP content length but it doesn't work
**URL:** https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861
**Category:** Elasticsearch
**Created:** [August 17, 2018, 9:22am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861 "2018-08-17T09:22:41Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@shwetha\_thakur](https://discuss.elastic.co/u/shwetha_thakur)
#### Post date: [August 17, 2018, 9:22am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/1 "2018-08-17T09:22:41Z")
I am using Elasticsearch 6.3 latest version. I am not able to bulk index a JSON file with size greater than 100MB. when I run the below command to bulk index, there will be no error but the index will not be created. I tried modifying HTTP content\_length, but doesn't work. Is there any way to store file size greater than 100MB?
Command which I run to bulk index:
curl -H "Content-Type: application/json" -XPOST "localhost:9200/bank/\_doc/\_bulk?pretty&refresh" --data-binary "@accounts.json"
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 17, 2018, 9:49am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/2 "2018-08-17T09:49:02Z")
A common guideline is to keep the size of each bill request to around 5MB in size. Sending larger bulk requests than that does not necessarily result in better performance.
---
### Author:  [@shwetha\_thakur](https://discuss.elastic.co/u/shwetha_thakur)
#### Post date: [August 17, 2018, 10:28am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/3 "2018-08-17T10:28:19Z")
Each bill request as in? what else is the way to store large amount of data in an index?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 17, 2018, 10:32am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/4 "2018-08-17T10:32:26Z")
Are you storing lots of small documents or very large documents?
---
### Author:  [@shwetha\_thakur](https://discuss.elastic.co/u/shwetha_thakur)
#### Post date: [August 17, 2018, 10:39am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/5 "2018-08-17T10:39:14Z")
small documents
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 17, 2018, 10:42am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/6 "2018-08-17T10:42:15Z")
Then break it up into multiple smaller bulk requests. Bulk requests are used to send groups of documents in a single request but very rarely all documents.
---
### Author:  [@shwetha\_thakur](https://discuss.elastic.co/u/shwetha_thakur)
#### Post date: [August 17, 2018, 10:46am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/7 "2018-08-17T10:46:34Z")
So you trying to say i should bulk index files each less than 100mb? Like I indexed a file in which 51,000 small documents are there. It's size is 95MB. It gets bulk indexed. If i try to add another field in each 51,000 documents, the total file size exceeds 100MB and doesn't get indexed.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 17, 2018, 10:48am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/8 "2018-08-17T10:48:43Z")
Try to keep the size of each request to around 5MB. If you use a tool like Logstash it will do this for you.
---
### Author:  [@shwetha\_thakur](https://discuss.elastic.co/u/shwetha_thakur)
#### Post date: [August 17, 2018, 10:50am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/9 "2018-08-17T10:50:11Z")
Each file size to 5MB or each document to 5MB?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 17, 2018, 10:52am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/10 "2018-08-17T10:52:34Z")
If you have small documents you should try to keep the request size around that level.
---
### Author:  [@shwetha\_thakur](https://discuss.elastic.co/u/shwetha_thakur)
#### Post date: [August 17, 2018, 11:28am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/11 "2018-08-17T11:28:44Z")
It doesn't seem to be relevant to bulk index 5MB files again and again. There are many e-commerce websites who use elasticsearch. They definitely store GBs of data. There should be a way.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 17, 2018, 12:15pm UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/12 "2018-08-17T12:15:06Z")
What does the bulk size have to do with how much data you store in the cluster??
Even if I just upload 1000 documents per bulk request, eventually the cluster will hold millions or billions of documents.
---
### Author:  [@shwetha\_thakur](https://discuss.elastic.co/u/shwetha_thakur)
#### Post date: [August 17, 2018, 12:27pm UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/13 "2018-08-17T12:27:24Z")
Okay. Thanks for your reply.
---
### Author:  [@neergttocsdivad](https://discuss.elastic.co/u/neergttocsdivad)
#### Post date: [August 21, 2018, 10:58am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/14 "2018-08-21T10:58:51Z")
have you looked at " Loading Wikipedia's Search Index For Testing"
> **[Loading Wikipedia's Search Index For Testing](https://www.elastic.co/blog/loading-wikipedia)**
>
> Load Wikipedia's real search index and experiment with it!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 18, 2018, 10:58am UTC](https://discuss.elastic.co/t/elastic-elasticsearch-cannot-bulk-index-a-json-file-greater-than-100mb-in-elasticsearch-tried-changing-http-content-length-but-it-doesnt-work/144861/15 "2018-09-18T10:58:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.