# SearchScroll hangs when dealing with a big amount of hits

**URL:** <https://discuss.elastic.co/t/searchscroll-hangs-when-dealing-with-a-big-amount-of-hits/7684>\
**Category:** Elasticsearch\
**Created:** [May 14, 2012, 1:28pm UTC](https://discuss.elastic.co/t/searchscroll-hangs-when-dealing-with-a-big-amount-of-hits/7684 "2012-05-14T13:28:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![piotrs](https://avatars.discourse-cdn.com/v4/letter/p/ee59a6/32.png) [@piotrs](https://discuss.elastic.co/u/piotrs)\
**Post date:** [May 14, 2012, 1:28pm UTC](https://discuss.elastic.co/t/searchscroll-hangs-when-dealing-with-a-big-amount-of-hits/7684/1 "2012-05-14T13:28:50Z")

</div>

Hi,

I'm using Java API to index/search in elasticsearch. I noticed that  
when search results are extremely big (for instance more than 200.000  
hits), scroll hangs after it has iterated about 50.000 records.

For instance I have following code to produce a search response:

SearchResponse scrollResp =  
client.prepareSearch().setIndices(indexName).setTypes(ES\_TYPE)  
.setSearchType(SearchType.SCAN).setQuery(buildQuery)  
.setScroll(new TimeValue(1000))  
.setSize(1000).execute().actionGet();

and then I do following to scroll through the result

while (true) {  
scrollResp =  
client.prepareSearchScroll(scrollResp.getScrollId()).setScroll(  
new TimeValue(1000)).execute().actionGet();  
boolean hitsRead = false;  
for (SearchHit hit : scrollResp.getHits()) {  
hitsRead = true;  
def sourceMap = hit.getSource();  
sourceMap.\_id = hit.getId();  
resultSet.result.add(sourceMap);  
}  
if (!hitsRead) {  
break;  
}  
}

The prepareSearchScroll() call tends to hang or even the for loop when  
reading the next record. Can't elasticsearch handle such big amount of  
hits and should the search be more restricted to produce less amount  
of hits?

br, Piotr

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [May 15, 2012, 8:46pm UTC](https://discuss.elastic.co/t/searchscroll-hangs-when-dealing-with-a-big-amount-of-hits/7684/2 "2012-05-15T20:46:00Z")

</div>

Scan is aimed at scrolling large amount of data. Do you see any failures  
the logs in the cluster nodes? Iterating through the hits  
will definitely not hang, since all are already represented in the search  
response, maybe your client does not have enough mem allocated to it to do  
the scrolling? Try and use a smaller size (like 100), see if it helps.

On Mon, May 14, 2012 at 4:28 PM, piotrs [piotr.skawinski@gmail.com](mailto:piotr.skawinski@gmail.com) wrote:

> Hi,
> 
> I'm using Java API to index/search in elasticsearch. I noticed that  
> when search results are extremely big (for instance more than 200.000  
> hits), scroll hangs after it has iterated about 50.000 records.
> 
> For instance I have following code to produce a search response:
> 
> SearchResponse scrollResp =  
> client.prepareSearch().setIndices(indexName).setTypes(ES\_TYPE)  
> .setSearchType(SearchType.SCAN).setQuery(buildQuery)  
> .setScroll(new TimeValue(1000))  
> .setSize(1000).execute().actionGet();
> 
> and then I do following to scroll through the result
> 
> while (true) {  
> scrollResp =  
> client.prepareSearchScroll(scrollResp.getScrollId()).setScroll(  
> new TimeValue(1000)).execute().actionGet();  
> boolean hitsRead = false;  
> for (SearchHit hit : scrollResp.getHits()) {  
> hitsRead = true;  
> def sourceMap = hit.getSource();  
> sourceMap.\_id = hit.getId();  
> resultSet.result.add(sourceMap);  
> }  
> if (!hitsRead) {  
> break;  
> }  
> }
> 
> The prepareSearchScroll() call tends to hang or even the for loop when  
> reading the next record. Can't elasticsearch handle such big amount of  
> hits and should the search be more restricted to produce less amount  
> of hits?
> 
> br, Piotr

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:28am UTC](https://discuss.elastic.co/t/searchscroll-hangs-when-dealing-with-a-big-amount-of-hits/7684/3 "2017-07-06T03:28:40Z")

</div>