# ScrollId Timeout **URL:** \ **Category:** Elasticsearch\ **Created:** [June 15, 2011, 11:53pm UTC](https://discuss.elastic.co/t/scrollid-timeout/4634 "2011-06-15T23:53:52Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 15, 2011, 11:53pm UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/1 "2011-06-15T23:53:52Z")
Is it necessary to pass the scrollId timeout value in each subsequent search scroll request? As in the example at [http://www.elasticsearch.org/guide/reference/api/search/search-type.html](http://www.elasticsearch.org/guide/reference/api/search/search-type.html) ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 16, 2011, 3:38am UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/2 "2011-06-16T03:38:13Z")
On Wed, 2011-06-15 at 19:53 -0400, James Cook wrote: > Is it necessary to pass the scrollId timeout value in each subsequent > search scroll request? The scroll timeout and the new \_scroll\_id from the previous search or scroll request. The final request will return zero hits, which is how you know that you're done. clint ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 16, 2011, 6:37am UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/3 "2011-06-16T06:37:20Z")
Hi Clinton, It doesn't seem to work that way. At least anecdotally, I don't have to pass the timeout value in subsequent search scroll requests. Perhaps this means that my "cursor" will time out based on the amount of time that has elapsed from the initial scroll request, but I was checking to see if this is the case. I wish the last result did return 0 hits, but instead it is currently (0.16.2) throwing an exception. > > > When using the \_scroll search to copy from one index to another I get an excepti…on\*\\\* for the last \_scroll search but all objects are successfully fetched + indexed (so not a real issue ;)). Or is my condition if (currentResults == 0) wrong but in the tutorial I read 'The “exit” point from the scrolling process is when no hits are returned back.'? > > Java code for the copying is: > > \`\`\` > SearchRequestBuilder srb = client.prepareSearch(fromIndex). > setVersion(true). > setQuery(QueryBuilders.matchAllQuery()).setSize(hitsPerPage). > setSearchType(SearchType.SCAN). > setScroll(TimeValue.timeValueMinutes(keepTime)); > if (additionalFilter != null) > srb.setFilter(additionalFilter); > SearchResponse rsp = srb.execute().actionGet(); > > try { > long total = rsp.hits().totalHits(); > String scrollId = rsp.scrollId(); > int collectedResults = 0; > while (true) { > rsp = client.prepareSearchScroll(scrollId). > setScroll(TimeValue.timeValueMinutes(keepTime)).execute().actionGet(); > long currentResults = rsp.hits().hits().length; > if (currentResults == 0) > break; > > // convert rsp into java objects > Collection\ objs = createObj.collectObjects(rsp); > // do bulk indexing of the grabbed objects > int failed = bulkUpdate(objs, intoIndex, false, false).size(); > // trying to enable flushing to avoid memory issues on the server side? > flush(intoIndex); > collectedResults += currentResults; > } > } catch (Exception ex) { > logger.error("Failed to copy data from index " + fromIndex + " into " + intoIndex + ".", ex); > } > > \----------------------------------------- > public Collection\ bulkUpdate(Collection\ objects, > String indexName, boolean refresh, boolean enableVersioning) { > > BulkRequestBuilder brb = client.prepareBulk(); > // this works differently then the direct call to refresh!? maybe refresh is not async? > // brb.setRefresh(refresh); > for (T o : objects) { > if (o.getId() == null) { > logger.warn("Skipped object without id when bulkUpdate:" + o); > continue; > } > > try { > XContentBuilder source = createDoc(o); > IndexRequest indexReq = Requests.indexRequest(indexName).type(getIndexType()).id(o.getId()).source(source); > if (enableVersioning) > indexReq.version(o.getVersion()); > > brb.add(indexReq); > } catch (IOException ex) { > logger.warn("Cannot add object:" + o + " to bulkIndexing action." + ex.getMessage()); > } > } > if (brb.numberOfActions() \> 0) { > BulkResponse rsp = brb.execute().actionGet(); > if (rsp.hasFailures()) { > List\ list = new ArrayList\(rsp.items().length); > for (BulkItemResponse br : rsp.items()) { > list.add(br.itemId()); > } > return list; > } > if (refresh) > refresh(indexName); > } > > return Collections.emptyList(); > } > \`\`\` > > \*\* > > \`\`\` > org.elasticsearch.transport.RemoteTransportException: \[Super Rabbit\]\[inet\[/127.0.0.1:9300\]\]\[indices/searchScroll\] > Caused by: org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase \[fetch\], \[reduce\] ; shardFailures {SearchContextMissingException\[No search context found for id \[204864\]\]}{SearchContextMissingException\[No search context found for id \[204865\]\]} > at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.finishHim(TransportSearchScrollScanAction.java:209) > at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.access$1300(TransportSearchScrollScanAction.java:80) > at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction$3.onFailure(TransportSearchScrollScanAction.java:199) > at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteScan(SearchServiceTransportAction.java:377) > at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.executePhase(TransportSearchScrollScanAction.java:184) > at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.access$700(TransportSearchScrollScanAction.java:80) > at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction$2.run(TransportSearchScrollScanAction.java:157) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.IndexOutOfBoundsException: index (0) must be less than size (0) > at org.elasticsearch.common.base.Preconditions.checkElementIndex(Preconditions.java:301) > at org.elasticsearch.common.base.Preconditions.checkElementIndex(Preconditions.java:280) > at org.elasticsearch.common.collect.Iterables.get(Iterables.java:649) > at org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:259) > at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.innerFinishHim(TransportSearchScrollScanAction.java:232) > at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.finishHim(TransportSearchScrollScanAction.java:207) > ... 9 more > \`\`\` -- jim On Wed, Jun 15, 2011 at 11:38 PM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote: > On Wed, 2011-06-15 at 19:53 -0400, James Cook wrote: > > > Is it necessary to pass the scrollId timeout value in each subsequent > > search scroll request? > > The scroll timeout and the new \_scroll\_id from the previous search or > scroll request. > > The final request will return zero hits, which is how you know that > you're done. > > clint ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 16, 2011, 6:50am UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/4 "2011-06-16T06:50:21Z")
Hi James > It doesn't seem to work that way. At least anecdotally, I don't have > to pass the timeout value in subsequent search scroll requests. > I wish the last result did return 0 hits, but instead it is currently > (0.16.2) throwing an exception. > [Scroll search always throws IndexOutOfBoundsException on last iteration · Issue #1008 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1008) Is the fact that you're not passing the timeout not the reason that you are seeing the exception? I use scrolling as I described, and it works without any errors. Note: if you don't pass the timeout, you may still see a few successful scroll results, but it won't last 😉 clint ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 16, 2011, 7:15am UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/5 "2011-06-16T07:15:57Z")
I attached a simple gist to the issue to recreate. If it succeeds for you, perhaps there is a config parameter which is different between our set ups which has some impact on the results? So, the timeout value needs to be constantly supplied on each successive search. I can't think of a use case where that is a helpful feature. Couldn't each time a scroll\_id is referenced, it updates its TTL with the original value. I suppose passing the timeout each time only makes sense if: ``` a) The TTL needs to change while retrieving pages of results, or b) ES doesn't have a way of knowing what the original TTL was. ``` Either way, it is a simple workaround even though it is a bit strange. -- jim On Thu, Jun 16, 2011 at 2:50 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote: > Hi James > > > It doesn't seem to work that way. At least anecdotally, I don't have > > to pass the timeout value in subsequent search scroll requests. > > > I wish the last result did return 0 hits, but instead it is currently > > (0.16.2) throwing an exception. > > [Scroll search always throws IndexOutOfBoundsException on last iteration · Issue #1008 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1008) > > Is the fact that you're not passing the timeout not the reason that you > are seeing the exception? > > I use scrolling as I described, and it works without any errors. Note: > if you don't pass the timeout, you may still see a few successful scroll > results, but it won't last 😉 > > clint ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 16, 2011, 7:31am UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/6 "2011-06-16T07:31:09Z")
Hi James On Thu, 2011-06-16 at 03:15 -0400, James Cook wrote: > I attached a simple gist to the issue to recreate. If it succeeds for > you, perhaps there is a config parameter which is different between > our set ups which has some impact on the results? I've got no special config Here's an example of a scrolled search which works for me on 0.16.2: > clint ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 16, 2011, 1:44pm UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/7 "2011-06-16T13:44:59Z")
Thanks clinton. Were you able to duplicate my recreation and the exception? On Jun 16, 2011 3:31 AM, "Clinton Gormley" [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote: > Hi James > > On Thu, 2011-06-16 at 03:15 -0400, James Cook wrote: > > > I attached a simple gist to the issue to recreate. If it succeeds for > > you, perhaps there is a config parameter which is different between > > our set ups which has some impact on the results? > > I've got no special config > > Here's an example of a scrolled search which works for me on 0.16.2: > > [Scrolled search · GitHub](https://gist.github.com/1028836) > > clint ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 16, 2011, 2:21pm UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/8 "2011-06-16T14:21:23Z")
Hi James On Thu, 2011-06-16 at 09:44 -0400, James Cook wrote: > Thanks clinton. Were you able to duplicate my recreation and the > exception? I don't use the Java API I'm afraid (or Java for that matter), so no 🙂 but if you look through the curl script that i linked to, you can check that you're doing the same steps that i did, and if there is a difference, then that's probably where the issue is. if there isn't, well then it may be a bug clint > > [Scrolled search · GitHub](https://gist.github.com/1028836) > > > > clint ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 16, 2011, 3:30pm UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/9 "2011-06-16T15:30:57Z")
It's not a Java recreation. It's Curl. Check out: [Scroll search always throws IndexOutOfBoundsException on last iteration · Issue #1008 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1008) It's very simple: curl -XPOST '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{ "user": "kimchy" }' curl -XGET 'localhost:9200/\_search?search\_type=scan&scroll=5m&pretty=true' -d '{ "query" : { "term": {"user":"kimchy"} } }' # get scrollID curl -GET 'localhost:9200/\_search/scroll?scroll=5m&pretty=true' -d '' # returns 1 hit curl -GET 'localhost:9200/\_search/scroll?scroll=5m&pretty=true' -d '' # throws exception On Thu, Jun 16, 2011 at 10:21 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote: > Hi James > > On Thu, 2011-06-16 at 09:44 -0400, James Cook wrote: > > > Thanks clinton. Were you able to duplicate my recreation and the > > exception? > > I don't use the Java API I'm afraid (or Java for that matter), so no 🙂 > > but if you look through the curl script that i linked to, you can check > that you're doing the same steps that i did, and if there is a > difference, then that's probably where the issue is. > > if there isn't, well then it may be a bug > > clint > > > > [Scrolled search · GitHub](https://gist.github.com/1028836) > > > > > > clint ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 16, 2011, 3:53pm UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/10 "2011-06-16T15:53:06Z")
Sorry - I'm flu ridden - I missed that: > # get scrollID > curl -GET 'localhost:9200/\_search/scroll?scroll=5m&pretty=true' -d > '' > > # returns 1 hit > curl -GET 'localhost:9200/\_search/scroll?scroll=5m&pretty=true' -d > '' Which scroll ID are you passing to the last statement? The scroll ID from the search? Or the scroll ID from the previous scroll request? It should be the latter clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 16, 2011, 4:22pm UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/11 "2011-06-16T16:22:12Z")
James I will check you case. Providing the scroll parameter (with the timeout) means that you want to continue scrolling the request. Not passing it means that you don't. When scrolling, you need to make sure that you pass the scroll id you got from the previous response to the next scroll request. On Thursday, June 16, 2011 at 6:53 PM, Clinton Gormley wrote: > Sorry - I'm flu ridden - I missed that: > > > # get scrollID > > > curl -GET 'localhost:9200/\_search/scroll?scroll=5m&pretty=true' -d > > '' > > > > # returns 1 hit > > > curl -GET 'localhost:9200/\_search/scroll?scroll=5m&pretty=true' -d > > '' > > Which scroll ID are you passing to the last statement? The scroll ID > > from the search? Or the scroll ID from the previous scroll request? > > It should be the latter > > clint ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 16, 2011, 5:47pm UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/12 "2011-06-16T17:47:23Z")
Thanks Shay and Clinton. That must be the problem. I have been using the scroll\_id from the very first "setup" request to make subsequent requests. I'll add a comment to the issue if this is the case. --- jim James I will check you case. Providing the scroll parameter (with the > timeout) means that you want to continue scrolling the request. Not passing > it means that you don't. When scrolling, you need to make sure that you pass > the scroll id you got from the previous response to the next scroll request. > > On Thursday, June 16, 2011 at 6:53 PM, Clinton Gormley wrote: > > Sorry - I'm flu ridden - I missed that: > > # get scrollID > > curl -GET 'localhost:9200/\_search/scroll?scroll=5m&pretty=true' -d > '' > > # returns 1 hit > > curl -GET 'localhost:9200/\_search/scroll?scroll=5m&pretty=true' -d > '' > > Which scroll ID are you passing to the last statement? The scroll ID > from the search? Or the scroll ID from the previous scroll request? > > It should be the latter > > clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:03am UTC](https://discuss.elastic.co/t/scrollid-timeout/4634/13 "2017-07-06T04:03:18Z")