{
{ "appendContent", document.DocumentContent }
}
```
};
var updateRes = _elasticClient.Update(updateRequest);
```
We are now consistently getting java heaps memory exception when trying to index more than 400 MB using above script.
We are hoping for the script above to add chunk by chunk into the index rather than load all the previous chunks and then append the next chunk. Because of the "load previous chunks before append the next chunk" behavior has defeated the purpose of our sending document in small chunks to index.
My first question is, is there any more that we could do to the above script to achieve what we want?
And if not the second question would be, can anyone recommend another way of doing this? that is ElasticSearch can index the same document chunk by chunk?
Please assist.
Regards
---
### Author:  [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)
#### Post date: [November 14, 2016, 12:03pm UTC](https://discuss.elastic.co/t/elastic-search-memory-heap-exception-when-trying-to-index-large-document-by-chunks/65924/2 "2016-11-14T12:03:15Z")
Hi @kiet.tran,
a 1 GB document size is way beyond what Elasticsearch is designed for. Do you really want to search within this document and then retrieve 1 GB if it matches? Here are a few options what you could do depending on your use case:
- Split the document into multiple smaller ones
- Store the 1GB file on the file system or in a blob store and extract only the data that you want (e.g. if this is a video file it makes sense only to store the metadata but not the actual movie in ES)
I see what you want to achieve with your script but documents in Elasticsearch are immutable so in fact, it will retrieve the data that you've sent so far and issue another index request for a new (version of the) document.
Daniel
---
### Author:  [@kiet.tran](https://discuss.elastic.co/u/kiet.tran)
#### Post date: [November 14, 2016, 9:46pm UTC](https://discuss.elastic.co/t/elastic-search-memory-heap-exception-when-trying-to-index-large-document-by-chunks/65924/3 "2016-11-14T21:46:31Z")
Hi @danielmitterdorfer ,
Thanks for the prompt response.
We are using highlighting to search for document so would expect to only to return the matched terms roughly 100 characters of where the term is found inside the document. so not expecting it to return 1GB.
The 1GB if is a .docx or .pdf extensions.
Will elastic search eventually provide indexing document by chunks in the future?
Cheers,
Kiet
---
### Author:  [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)
#### Post date: [November 15, 2016, 9:15am UTC](https://discuss.elastic.co/t/elastic-search-memory-heap-exception-when-trying-to-index-large-document-by-chunks/65924/4 "2016-11-15T09:15:03Z")
Hi @kiet.tran,
> [@kiet.tran](#):
>
> The 1GB if is a .docx or .pdf extensions.
That's pretty large documents. 🙂 I'd split each docx / pdf document then accordingly, e.g. one section = 1 Elasticsearch document. Several MB per (Elasticsearch) document are definitely not a problem but 1 GB is definitely beyond the sweet spot.
> [@kiet.tran](#):
>
> Will Elasticsearch eventually provide indexing document by chunks in the future?
I doubt it as this would require very significant (architectural) changes and Elasticsearch would be an entirely different product then. But never say never.
Daniel
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 13, 2016, 9:15am UTC](https://discuss.elastic.co/t/elastic-search-memory-heap-exception-when-trying-to-index-large-document-by-chunks/65924/5 "2016-12-13T09:15:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.