# ES OutOfMemoryError while indexing a large number of attachments **URL:** \ **Category:** Elasticsearch\ **Created:** [March 30, 2012, 4:03pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182 "2012-03-30T16:03:57Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [March 30, 2012, 4:03pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/1 "2012-03-30T16:03:57Z")
I've been working to stabilze a process where I need to index ~16K attachments. The following is the latest result: > I'm running ES with ES\_HEAP\_SIZE=2g but still running into an issue of running out of heap after indexing ~5K attachments. The attachments range in size from a few KB to no more than 20MB. The indexing process currently uses 2 workers to update already indexed documents by adding the attachments via the update API. See code here: > I'd like suggestions on how to get this process to work without ES running out of memory. 2GB seems plenty for what I'm trying to do and giving more memory isn't really possible since the box only has 4GB. Thanks, Shane ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [March 30, 2012, 4:07pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/2 "2012-03-30T16:07:36Z")
I forgot to mention that I also get a shard failure around the same time the OutOfMemoryException occurs. On Friday, March 30, 2012 12:03:57 PM UTC-4, Shane Witbeck wrote: > I've been working to stabilze a process where I need to index ~16K > attachments. The following is the latest result: > > [ES failure after large number of attachments indexed · GitHub](https://gist.github.com/2252398) > > I'm running ES with ES\_HEAP\_SIZE=2g but still running into an issue of > running out of heap after indexing ~5K attachments. The attachments range > in size from a few KB to no more than 20MB. > > The indexing process currently uses 2 workers to update already indexed > documents by adding the attachments via the update API. See code here: > > [indexAttachment · GitHub](https://gist.github.com/2252482) > > I'd like suggestions on how to get this process to work without ES running > out of memory. 2GB seems plenty for what I'm trying to do and giving more > memory isn't really possible since the box only has 4GB. > > Thanks, > Shane ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 31, 2012, 9:18pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/3 "2012-03-31T21:18:04Z")
Is there a chance that you have a singel document that you end up adding many attachments to resulting in the OOM failure? Update API simply reads the current doc and then reindex all of it back. On Fri, Mar 30, 2012 at 7:07 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > I forgot to mention that I also get a shard failure around the same time > the OutOfMemoryException occurs. > > On Friday, March 30, 2012 12:03:57 PM UTC-4, Shane Witbeck wrote: > > > I've been working to stabilze a process where I need to index ~16K > > attachments. The following is the latest result: > > > > [https://gist.github.com/\*\*2252398](https://gist.github.com/**2252398) [https://gist.github.com/2252398](https://gist.github.com/2252398) > > > > I'm running ES with ES\_HEAP\_SIZE=2g but still running into an issue of > > running out of heap after indexing ~5K attachments. The attachments range > > in size from a few KB to no more than 20MB. > > > > The indexing process currently uses 2 workers to update already indexed > > documents by adding the attachments via the update API. See code here: > > > > [https://gist.github.com/\*\*2252482](https://gist.github.com/**2252482) [https://gist.github.com/2252482](https://gist.github.com/2252482) > > > > I'd like suggestions on how to get this process to work without ES > > running out of memory. 2GB seems plenty for what I'm trying to do and > > giving more memory isn't really possible since the box only has 4GB. > > > > Thanks, > > Shane ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [March 31, 2012, 11:36pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/4 "2012-03-31T23:36:37Z")
Yes, each document may have several attachments associated with it. ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [April 2, 2012, 9:18pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/5 "2012-04-02T21:18:30Z")
Given the scenario I've outlined, does it make more sense to put attachments in their own index? It seems I've hit a limitation of the attachment plugin with the limited amount of RAM that I have and the potential of several mutli-MB attachments per document. I'm also curious if you think increasing the amount of RAM on the machines would help in this case? I have just the one index and was hoping to avoid creating another index for attachments but if this is the way to go what would be the best way to associate them? Thanks, Shane On Saturday, March 31, 2012 7:36:37 PM UTC-4, Shane Witbeck wrote: > Yes, each document may have several attachments associated with it. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 3, 2012, 2:32pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/6 "2012-04-03T14:32:10Z")
Yea, probably breaking down the attachments to their own docs make more sense. On Tue, Apr 3, 2012 at 12:18 AM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > Given the scenario I've outlined, does it make more sense to put > attachments in their own index? It seems I've hit a limitation of the > attachment plugin with the limited amount of RAM that I have and the > potential of several mutli-MB attachments per document. I'm also curious if > you think increasing the amount of RAM on the machines would help in this > case? > > I have just the one index and was hoping to avoid creating another index > for attachments but if this is the way to go what would be the best way to > associate them? > > Thanks, > Shane > > On Saturday, March 31, 2012 7:36:37 PM UTC-4, Shane Witbeck wrote: > > > Yes, each document may have several attachments associated with it. ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [April 3, 2012, 2:40pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/7 "2012-04-03T14:40:19Z")
I don't have experience (yet) with searching on more than one index at a time. Is the multisearch API ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/multi-search.html)) what I need here or is there some way of associating one index with another? Thanks On Tuesday, April 3, 2012 10:32:10 AM UTC-4, kimchy wrote: > Yea, probably breaking down the attachments to their own docs make more > sense. > > On Tue, Apr 3, 2012 at 12:18 AM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > Given the scenario I've outlined, does it make more sense to put > > attachments in their own index? It seems I've hit a limitation of the > > attachment plugin with the limited amount of RAM that I have and the > > potential of several mutli-MB attachments per document. I'm also curious if > > you think increasing the amount of RAM on the machines would help in this > > case? > > > > I have just the one index and was hoping to avoid creating another index > > for attachments but if this is the way to go what would be the best way to > > associate them? > > > > Thanks, > > Shane > > > > On Saturday, March 31, 2012 7:36:37 PM UTC-4, Shane Witbeck wrote: > > > > > Yes, each document may have several attachments associated with it. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 3, 2012, 3:05pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/8 "2012-04-03T15:05:07Z")
You just specify the two index names in the URI, or if you use the Java API, specify both indices in the search API. On Tue, Apr 3, 2012 at 5:40 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > I don't have experience (yet) with searching on more than one index at a > time. Is the multisearch API ( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/multi-search.html)) what > I need here or is there some way of associating one index with another? > > Thanks > > On Tuesday, April 3, 2012 10:32:10 AM UTC-4, kimchy wrote: > > > Yea, probably breaking down the attachments to their own docs make more > > sense. > > > > On Tue, Apr 3, 2012 at 12:18 AM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > > > Given the scenario I've outlined, does it make more sense to put > > > attachments in their own index? It seems I've hit a limitation of the > > > attachment plugin with the limited amount of RAM that I have and the > > > potential of several mutli-MB attachments per document. I'm also curious if > > > you think increasing the amount of RAM on the machines would help in this > > > case? > > > > > > I have just the one index and was hoping to avoid creating another index > > > for attachments but if this is the way to go what would be the best way to > > > associate them? > > > > > > Thanks, > > > Shane > > > > > > On Saturday, March 31, 2012 7:36:37 PM UTC-4, Shane Witbeck wrote: > > > > > > > Yes, each document may have several attachments associated with it. ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [April 3, 2012, 4:09pm UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/9 "2012-04-03T16:09:57Z")
My hunch is that Tika is running out of memory on you. Tika moved from using filesystem for temp storage to an in memory based approach for PDFs, which led to some out of memory issues on my side. Really, it is PDFBox under Tika that made the change and I believe tika 0.9 picked this up. You should be able to confirm by analyzing the heap dump. In Tika 1.1, you can control this behavior by passing in a file object instead of a stream. Best Regards, Paul On Apr 3, 9:05 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > You just specify the two index names in the URI, or if you use the Java > API, specify both indices in the search API. > > On Tue, Apr 3, 2012 at 5:40 PM, Shane Witbeck [sh...@digitalsanctum.com](mailto:sh...@digitalsanctum.com)wrote: > > > I don't have experience (yet) with searching on more than one index at a > > time. Is the multisearch API ( > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/multi-search.html)) what > > I need here or is there some way of associating one index with another? > > > Thanks > > > On Tuesday, April 3, 2012 10:32:10 AM UTC-4, kimchy wrote: > > > > Yea, probably breaking down the attachments to their own docs make more > > > sense. > > > > On Tue, Apr 3, 2012 at 12:18 AM, Shane Witbeck [sh...@digitalsanctum.com](mailto:sh...@digitalsanctum.com)wrote: > > > > > Given the scenario I've outlined, does it make more sense to put > > > > attachments in their own index? It seems I've hit a limitation of the > > > > attachment plugin with the limited amount of RAM that I have and the > > > > potential of several mutli-MB attachments per document. I'm also curious if > > > > you think increasing the amount of RAM on the machines would help in this > > > > case? > > > > > I have just the one index and was hoping to avoid creating another index > > > > for attachments but if this is the way to go what would be the best way to > > > > associate them? > > > > > Thanks, > > > > Shane > > > > > On Saturday, March 31, 2012 7:36:37 PM UTC-4, Shane Witbeck wrote: > > > > > > Yes, each document may have several attachments associated with it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:33am UTC](https://discuss.elastic.co/t/es-outofmemoryerror-while-indexing-a-large-number-of-attachments/7182/10 "2017-07-06T03:33:48Z")