# Iterator over whole result set (Java API)

**URL:** <https://discuss.elastic.co/t/iterator-over-whole-result-set-java-api/3203>\
**Category:** Elasticsearch\
**Created:** [August 10, 2010, 2:12pm UTC](https://discuss.elastic.co/t/iterator-over-whole-result-set-java-api/3203 "2010-08-10T14:12:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\
**Post date:** [August 10, 2010, 2:12pm UTC](https://discuss.elastic.co/t/iterator-over-whole-result-set-java-api/3203/1 "2010-08-10T14:12:39Z")

</div>

Hi,  
I need to iterate over whole result set, meaning all of  
hits.getTotalHits() results from response (for some query). Required  
'size' is not defined before I execute query (I need all matches), so I  
can not use query.setSize(N).

What is the best way to get 'total hits' iterator and iterate over  
potentiality big results set?

I could execute 'count query' to get the number of matches for that  
query and use query.setSize(count query result) in main query, but I'm  
not sure this approach scales very well (I need to be able to iterate  
over possibly millions of documents)

Thanks,  
Tomislav

---

<div class="post-metadata">

**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\
**Post date:** [August 10, 2010, 2:47pm UTC](https://discuss.elastic.co/t/iterator-over-whole-result-set-java-api/3203/2 "2010-08-10T14:47:21Z")

</div>

I think you can use the scrolling API but it is expensive in terms of  
performance.  
[http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/scroll/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/scroll/)

Tal

On Aug 10, 5:12 pm, Tomislav Poljak [tpol...@gmail.com](mailto:tpol...@gmail.com) wrote:

> Hi,  
> I need to iterate over whole result set, meaning all of  
> hits.getTotalHits() results from response (for some query). Required  
> 'size' is not defined before I execute query (I need all matches), so I  
> can not use query.setSize(N).
> 
> What is the best way to get 'total hits' iterator and iterate over  
> potentiality big results set?
> 
> I could execute 'count query' to get the number of matches for that  
> query and use query.setSize(count query result) in main query, but I'm  
> not sure this approach scales very well (I need to be able to iterate  
> over possibly millions of documents)
> 
> Thanks,  
> Tomislav

---

<div class="post-metadata">

**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\
**Post date:** [August 10, 2010, 3:24pm UTC](https://discuss.elastic.co/t/iterator-over-whole-result-set-java-api/3203/3 "2010-08-10T15:24:39Z")

</div>

Hi Tal,  
can you give me an example of how can 'scroll' be used with ES Java API?

For example, in code like below:

SearchResponse response = client.search(

searchRequest(index).types(type).searchType(SearchType.QUERY\_AND\_FETCH).source(

searchSource().query(queryString(queryStr)).explain(false))).actionGet();

```
    SearchHits hits = response.getHits();

    Iterator<SearchHit> iterator = hits.iterator();

    while (iterator.hasNext()) {
        SearchHit searchHit = (SearchHit) iterator.next();
        //need all hits here
    }

```

Will I get all hits with scroll param?

Thanks,  
Tomislav

On Tue, 2010-08-10 at 07:47 -0700, Tal wrote:

> I think you can use the scrolling API but it is expensive in terms of  
> performance.  
> [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/scroll/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/scroll/)
> 
> Tal
> 
> On Aug 10, 5:12 pm, Tomislav Poljak [tpol...@gmail.com](mailto:tpol...@gmail.com) wrote:
> 
> > Hi,  
> > I need to iterate over whole result set, meaning all of  
> > hits.getTotalHits() results from response (for some query). Required  
> > 'size' is not defined before I execute query (I need all matches), so I  
> > can not use query.setSize(N).
> > 
> > What is the best way to get 'total hits' iterator and iterate over  
> > potentiality big results set?
> > 
> > I could execute 'count query' to get the number of matches for that  
> > query and use query.setSize(count query result) in main query, but I'm  
> > not sure this approach scales very well (I need to be able to iterate  
> > over possibly millions of documents)
> > 
> > Thanks,  
> > Tomislav

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 10, 2010, 5:45pm UTC](https://discuss.elastic.co/t/iterator-over-whole-result-set-java-api/3203/4 "2010-08-10T17:45:10Z")

</div>

Hi,

here is a gist with an example: [gist:517664 · GitHub](http://gist.github.com/517664).

-shay.banon

On Tue, Aug 10, 2010 at 6:24 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com) wrote:

> Hi Tal,  
> can you give me an example of how can 'scroll' be used with ES Java API?
> 
> For example, in code like below:
> 
> SearchResponse response = client.search(
> 
> searchRequest(index).types(type).searchType(SearchType.QUERY\_AND\_FETCH).source(
> 
> searchSource().query(queryString(queryStr)).explain(false))).actionGet();
> 
> ```
> SearchHits hits = response.getHits();
> 
> Iterator<SearchHit> iterator = hits.iterator();
> 
> while (iterator.hasNext()) {
> SearchHit searchHit = (SearchHit) iterator.next();
> //need all hits here
> }
> 
> ```
> 
> Will I get all hits with scroll param?
> 
> Thanks,  
> Tomislav
> 
> On Tue, 2010-08-10 at 07:47 -0700, Tal wrote:
> 
> > I think you can use the scrolling API but it is expensive in terms of  
> > performance.  
> > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/scroll/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/scroll/)
> > 
> > Tal
> > 
> > On Aug 10, 5:12 pm, Tomislav Poljak [tpol...@gmail.com](mailto:tpol...@gmail.com) wrote:
> > 
> > > Hi,  
> > > I need to iterate over whole result set, meaning all of  
> > > hits.getTotalHits() results from response (for some query). Required  
> > > 'size' is not defined before I execute query (I need all matches), so I  
> > > can not use query.setSize(N).
> > > 
> > > What is the best way to get 'total hits' iterator and iterate over  
> > > potentiality big results set?
> > > 
> > > I could execute 'count query' to get the number of matches for that  
> > > query and use query.setSize(count query result) in main query, but I'm  
> > > not sure this approach scales very well (I need to be able to iterate  
> > > over possibly millions of documents)
> > > 
> > > Thanks,  
> > > Tomislav

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 10, 2010, 5:47pm UTC](https://discuss.elastic.co/t/iterator-over-whole-result-set-java-api/3203/5 "2010-08-10T17:47:08Z")

</div>

One thing to remember though, scrolling is a heavy operation (its like  
a cursor in a database), and also you won't get duplicates or anything like  
that (actually, you won't see any changes done after you started the first  
scroll). So, it should not be heavily used.

-shay.banon

On Tue, Aug 10, 2010 at 8:45 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Hi,
> 
> here is a gist with an example: [gist:517664 · GitHub](http://gist.github.com/517664).
> 
> -shay.banon
> 
> On Tue, Aug 10, 2010 at 6:24 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)wrote:
> 
> > Hi Tal,  
> > can you give me an example of how can 'scroll' be used with ES Java API?
> > 
> > For example, in code like below:
> > 
> > SearchResponse response = client.search(
> > 
> > searchRequest(index).types(type).searchType(SearchType.QUERY\_AND\_FETCH).source(
> > 
> > searchSource().query(queryString(queryStr)).explain(false))).actionGet();
> > 
> > ```
> > SearchHits hits = response.getHits();
> > 
> > Iterator<SearchHit> iterator = hits.iterator();
> > 
> > while (iterator.hasNext()) {
> > SearchHit searchHit = (SearchHit) iterator.next();
> > //need all hits here
> > }
> > 
> > ```
> > 
> > Will I get all hits with scroll param?
> > 
> > Thanks,  
> > Tomislav
> > 
> > On Tue, 2010-08-10 at 07:47 -0700, Tal wrote:
> > 
> > > I think you can use the scrolling API but it is expensive in terms of  
> > > performance.  
> > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/scroll/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/scroll/)
> > > 
> > > Tal
> > > 
> > > On Aug 10, 5:12 pm, Tomislav Poljak [tpol...@gmail.com](mailto:tpol...@gmail.com) wrote:
> > > 
> > > > Hi,  
> > > > I need to iterate over whole result set, meaning all of  
> > > > hits.getTotalHits() results from response (for some query). Required  
> > > > 'size' is not defined before I execute query (I need all matches), so  
> > > > I  
> > > > can not use query.setSize(N).
> > > > 
> > > > What is the best way to get 'total hits' iterator and iterate over  
> > > > potentiality big results set?
> > > > 
> > > > I could execute 'count query' to get the number of matches for that  
> > > > query and use query.setSize(count query result) in main query, but I'm  
> > > > not sure this approach scales very well (I need to be able to iterate  
> > > > over possibly millions of documents)
> > > > 
> > > > Thanks,  
> > > > Tomislav

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:20am UTC](https://discuss.elastic.co/t/iterator-over-whole-result-set-java-api/3203/6 "2017-07-06T04:20:59Z")

</div>