# \[0.19.1\] OutOfMemoryError after setting a large result size **URL:** \ **Category:** Elasticsearch\ **Created:** [March 28, 2012, 1:26am UTC](https://discuss.elastic.co/t/0-19-1-outofmemoryerror-after-setting-a-large-result-size/7158 "2012-03-28T01:26:20Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [March 28, 2012, 1:26am UTC](https://discuss.elastic.co/t/0-19-1-outofmemoryerror-after-setting-a-large-result-size/7158/1 "2012-03-28T01:26:20Z")
SearchSourceBuilder search = new SearchSourceBuilder() .query(...).from(0).size(Integer.MAX\_VALUE); index.search(search).hits(); -\> Exception in thread "elasticsearch[search]-pool-47-thread-1" java.lang.OutOfMemoryError: Java heap space at org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java: 579) at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java: 317) I'd be less surprised if the index didn't just have a dozen small documents; looks like elasticsearch is preallocating a large array? ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [March 28, 2012, 2:00am UTC](https://discuss.elastic.co/t/0-19-1-outofmemoryerror-after-setting-a-large-result-size/7158/2 "2012-03-28T02:00:18Z")
My Lucene internals may be out of date, but if it's the same as a while back, the PriorityQueue used to hold the results is backed by an array with size . If you want all results, don't 'search', 'scroll' instead. On 28 March 2012 12:26, Eric Jain [eric.jain@gmail.com](mailto:eric.jain@gmail.com) wrote: > SearchSourceBuilder search = new SearchSourceBuilder() > .query(...).from(0).size(Integer.MAX\_VALUE); > index.search(search).hits(); > > -\> Exception in thread "elasticsearch[search]-pool-47-thread-1" > java.lang.OutOfMemoryError: Java heap space > at > > org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java: > 579) > at > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java: > 317) > > I'd be less surprised if the index didn't just have a dozen small > documents; looks like elasticsearch is preallocating a large array? ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [March 28, 2012, 6:40am UTC](https://discuss.elastic.co/t/0-19-1-outofmemoryerror-after-setting-a-large-result-size/7158/3 "2012-03-28T06:40:09Z")
On Mar 27, 7:00 pm, Paul Smith [tallpsm...@gmail.com](mailto:tallpsm...@gmail.com) wrote: > My Lucene internals may be out of date, but if it's the same as a while > back, the PriorityQueue used to hold the results is backed by an array with > size . > > If you want all results, don't 'search', 'scroll' instead. I can do that, but wouldn't it make sense to allocate no more than Math.min(size, numberOfDocuments)? ---
**Author:** ![Thomas\_Peuss](https://avatars.discourse-cdn.com/v4/letter/t/22d042/32.png) [@Thomas\_Peuss](https://discuss.elastic.co/u/Thomas_Peuss)\ **Post date:** [March 28, 2012, 6:56am UTC](https://discuss.elastic.co/t/0-19-1-outofmemoryerror-after-setting-a-large-result-size/7158/4 "2012-03-28T06:56:24Z")
Hi Eric! Am Mittwoch, 28. März 2012 08:40:09 UTC+2 schrieb Eric Jain: > On Mar 27, 7:00 pm, Paul Smith [tallpsm...@gmail.com](mailto:tallpsm...@gmail.com) wrote: > > > My Lucene internals may be out of date, but if it's the same as a while > > back, the PriorityQueue used to hold the results is backed by an array > > with > > size . > > > > If you want all results, don't 'search', 'scroll' instead. > > I can do that, but wouldn't it make sense to allocate no more than > Math.min(size, numberOfDocuments)? If you really need all documents of the index you should really have a look at the "scan/scroll"-API ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html)). If you know that your index will hold only a few hundred documents then why use MAX\_VALUE? Simple math like Math.min() can be dangerous because you allocate the array and then start to fill it with hits of the index. While you do that your index can become larger because of fresh documents arriving. This might be no problem in your case but we index 1000 docs/s. CU Thomas ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [March 28, 2012, 6:57am UTC](https://discuss.elastic.co/t/0-19-1-outofmemoryerror-after-setting-a-large-result-size/7158/5 "2012-03-28T06:57:31Z")
On Wednesday, 28 March 2012, Eric Jain [eric.jain@gmail.com](mailto:eric.jain@gmail.com) wrote: > On Mar 27, 7:00 pm, Paul Smith [tallpsm...@gmail.com](mailto:tallpsm...@gmail.com) wrote: > > > My Lucene internals may be out of date, but if it's the same as a while > > back, the PriorityQueue used to hold the results is backed by an array > > with > > size . > > > > If you want all results, don't 'search', 'scroll' instead. > > I can do that, but wouldn't it make sense to allocate no more than > Math.min(size, numberOfDocuments)? Lucene is optimizing for the common case of only the top X of hits being needed. It's more efficient for scoring sort to do it this way. Far less memory is used. How would you sort say a billion documents? Only documents that match the query/filters are passed through the priority queue for sorting and for common case of only needing top X we're talking a lot less overall comparisons and a lot less memory. The larger the hit size the more comparisons and memory. even if you did the math.min, for us with large indexes it would be allocating an array of huge size when we only want the first 25 most of the time. That's a waste. ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [March 28, 2012, 5:54pm UTC](https://discuss.elastic.co/t/0-19-1-outofmemoryerror-after-setting-a-large-result-size/7158/6 "2012-03-28T17:54:42Z")
On Tue, Mar 27, 2012 at 23:56, Thomas Peuss [thomas.peuss@nterra.com](mailto:thomas.peuss@nterra.com) wrote: > If you really need all documents of the index you should really have a look > at the "scan/scroll"-API > ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html)). > If you know that your index will hold only a few hundred documents then why > use MAX\_VALUE? > > Simple math like Math.min() can be dangerous because you allocate the array > and then start to fill it with hits of the index. While you do that your > index can become larger because of fresh documents arriving. This might be > no problem in your case but we index 1000 docs/s. Hadn't considered this case; I wouldn't mind if it was just new docs that were missing, but having some new docs showing up in place of older docs could indeed be confusing. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:34am UTC](https://discuss.elastic.co/t/0-19-1-outofmemoryerror-after-setting-a-large-result-size/7158/7 "2017-07-06T03:34:17Z")