# How to cover all conditions when fetch all data with scroll?
**URL:** https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461
**Category:** Elasticsearch
**Created:** [December 31, 2019, 3:22pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461 "2019-12-31T15:22:35Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@riverbuilding](https://discuss.elastic.co/u/riverbuilding)
#### Post date: [December 31, 2019, 3:22pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461/1 "2019-12-31T15:22:35Z")
based on this discuss:
> [@Scroll id is not changing while querying](https://discuss.elastic.co/t/scroll-id-is-not-changing-while-querying/106202):
>
> I written this code to test scroll api. As mentioned in docs the scroll id will change for every request. But in my case it is same for all requests. The initial search request and each subsequent scroll request returns a new \_scroll\_id — only the most recent \_scroll\_id should be used. Here is my code sample from elasticsearch import Elasticsearch es = Elasticsearch(['10.225.253.130:9200'], sniff\_on\_start=True) my\_dict = {} my\_dict['query'] = {} my\_dict['query']['match\_all'] = {} result…
the content I quote below:
Short answer: yes, if you have a single shard index (as seems to be in your case) - it is expected behavior, but it can happen even if you have multiple shards. Longer answer: the scroll basically contains a list of shards where your search is running plus information about how to find your scroll data on each shard. As you exhaust results from each shard, you will notice that the scroll id becomes shorter, because we no longer need to search these shards and therefore don't need to list them on scroll. But if only have one shard or all shards will get processed at the same time, your scroll id might never change. Saying this, I wouldn't rely on this behavior since it might change in the future and always copy scroll id from the previous response.
so my question is: considering the fact that scrollId may stay same under certain conditions, how to write code to fetch all data exactly once(in order to make things easier, let's exclude the modification action when fetch data)?
in my case, I have 1 shard index and it seems the below iteration loop is infinite:
while (searchHits != null && searchHits.length \> 0) {
...
}
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 31, 2019, 7:28pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461/2 "2019-12-31T19:28:52Z")
> [@riverbuilding](#):
>
> so my question is: considering the fact that scrollId may stay same under certain conditions, how to write code to fetch all data exactly once(in order to make things easier, let's exclude the modification action when fetch data)?
When used as documented, a scroll already returns each document once.
---
### Author:  [@riverbuilding](https://discuss.elastic.co/u/riverbuilding)
#### Post date: [December 31, 2019, 7:37pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461/3 "2019-12-31T19:37:47Z")
in my case, I have 1 shard index and it seems the below iteration loop is infinite:
while (searchHits != null && searchHits.length \> 0) {
...
}
how to deal with infinite loop if there is 1 shard? how to deal this situation I really met?
---
### Author:  [@riverbuilding](https://discuss.elastic.co/u/riverbuilding)
#### Post date: [January 2, 2020, 8:25pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461/4 "2020-01-02T20:25:28Z")
>
from this link, we are supposed to talk the issue here.
so let me add more info:
1. I do met infinite loop with the sample code in doc
2. I do have one shard, that's the reason I guess caused the infinite loop.
3. my ES version is 7.3.1
4. I use ES high level rest client
what else do u guys want to know?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 2, 2020, 10:00pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461/5 "2020-01-02T22:00:05Z")
The [code in the docs](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.3/java-rest-high-search-scroll.html#java-rest-high-search-scroll-example) terminates correctly after retrieving all documents. It's checked as part of the test suite. The number of shards isn't relevant. I am guessing you've altered this code somehow and now it's not working? You haven't really shared enough information to reproduce what you're seeing.
---
### Author:  [@riverbuilding](https://discuss.elastic.co/u/riverbuilding)
#### Post date: [January 2, 2020, 11:04pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461/6 "2020-01-02T23:04:21Z")
let me clarify my situation:
1. I have N documents in one index with one share only.
2. I used search-scroll-example but add this line: searchSourceBuilder.size(SCROLL\_SIZE);
3. here, SCROLL\_SIZE \> N
4. then I fall into infinite loop with same scrollId keep showing
what other info do u need?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 3, 2020, 12:53pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461/7 "2020-01-03T12:53:21Z")
I tried doing what you describe:
```diff
diff --git a/client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/SearchDocumentationIT.java b/client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/SearchDocumentationIT.java
index 995a50508fc..6fba679b66e 100644
--- a/client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/SearchDocumentationIT.java
+++ b/client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/SearchDocumentationIT.java
@@ -709,6 +709,7 @@ public class SearchDocumentationIT extends ESRestHighLevelClientTestCase {
searchRequest.scroll(scroll);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(matchQuery("title", "Elasticsearch"));
+ searchSourceBuilder.size(between(1, 10));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT); // <1>
```
In the test there are three docs, so this covers the cases where the size is both smaller and larger than the number of docs. It still passes.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 31, 2020, 12:53pm UTC](https://discuss.elastic.co/t/how-to-cover-all-conditions-when-fetch-all-data-with-scroll/213461/8 "2020-01-31T12:53:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.