ScrollId Timeout

James_Cook · June 15, 2011, 11:53pm

Is it necessary to pass the scrollId timeout value in each subsequent search
scroll request?

As in the example at
http://www.elasticsearch.org/guide/reference/api/search/search-type.html

Clinton_Gormley · June 16, 2011, 3:38am

On Wed, 2011-06-15 at 19:53 -0400, James Cook wrote:

Is it necessary to pass the scrollId timeout value in each subsequent
search scroll request?

The scroll timeout and the new _scroll_id from the previous search or
scroll request.

The final request will return zero hits, which is how you know that
you're done.

clint

James_Cook · June 16, 2011, 6:37am

Hi Clinton,

It doesn't seem to work that way. At least anecdotally, I don't have to pass
the timeout value in subsequent search scroll requests. Perhaps this means
that my "cursor" will time out based on the amount of time that has elapsed
from the initial scroll request, but I was checking to see if this is the
case.

I wish the last result did return 0 hits, but instead it is currently
(0.16.2) throwing an exception.

github.com/elastic/elasticsearch

Scroll search always throws IndexOutOfBoundsException on last iteration

opened 03:48PM - 08 Jun 11 UTC

closed 09:08AM - 20 Jun 11 UTC

karussell

When using the _scroll search to copy from one index to another I get an excepti…on*\* for the last _scroll search but all objects are successfully fetched + indexed (so not a real issue ;)). Or is my condition if (currentResults == 0) wrong but in the tutorial I read 'The “exit” point from the scrolling process is when no hits are returned back.'? Java code for the copying is: ``` SearchRequestBuilder srb = client.prepareSearch(fromIndex). setVersion(true). setQuery(QueryBuilders.matchAllQuery()).setSize(hitsPerPage). setSearchType(SearchType.SCAN). setScroll(TimeValue.timeValueMinutes(keepTime)); if (additionalFilter != null) srb.setFilter(additionalFilter); SearchResponse rsp = srb.execute().actionGet(); try { long total = rsp.hits().totalHits(); String scrollId = rsp.scrollId(); int collectedResults = 0; while (true) { rsp = client.prepareSearchScroll(scrollId). setScroll(TimeValue.timeValueMinutes(keepTime)).execute().actionGet(); long currentResults = rsp.hits().hits().length; if (currentResults == 0) break; // convert rsp into java objects Collection<T> objs = createObj.collectObjects(rsp); // do bulk indexing of the grabbed objects int failed = bulkUpdate(objs, intoIndex, false, false).size(); // trying to enable flushing to avoid memory issues on the server side? flush(intoIndex); collectedResults += currentResults; } } catch (Exception ex) { logger.error("Failed to copy data from index " + fromIndex + " into " + intoIndex + ".", ex); } ----------------------------------------- public Collection<Integer> bulkUpdate(Collection<T> objects, String indexName, boolean refresh, boolean enableVersioning) { BulkRequestBuilder brb = client.prepareBulk(); // this works differently then the direct call to refresh!? maybe refresh is not async? // brb.setRefresh(refresh); for (T o : objects) { if (o.getId() == null) { logger.warn("Skipped object without id when bulkUpdate:" + o); continue; } try { XContentBuilder source = createDoc(o); IndexRequest indexReq = Requests.indexRequest(indexName).type(getIndexType()).id(o.getId()).source(source); if (enableVersioning) indexReq.version(o.getVersion()); brb.add(indexReq); } catch (IOException ex) { logger.warn("Cannot add object:" + o + " to bulkIndexing action." + ex.getMessage()); } } if (brb.numberOfActions() > 0) { BulkResponse rsp = brb.execute().actionGet(); if (rsp.hasFailures()) { List<Integer> list = new ArrayList<Integer>(rsp.items().length); for (BulkItemResponse br : rsp.items()) { list.add(br.itemId()); } return list; } if (refresh) refresh(indexName); } return Collections.emptyList(); } ``` ** ``` org.elasticsearch.transport.RemoteTransportException: [Super Rabbit][inet[/127.0.0.1:9300]][indices/searchScroll] Caused by: org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [fetch], [reduce] ; shardFailures {SearchContextMissingException[No search context found for id [204864]]}{SearchContextMissingException[No search context found for id [204865]]} at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.finishHim(TransportSearchScrollScanAction.java:209) at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.access$1300(TransportSearchScrollScanAction.java:80) at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction$3.onFailure(TransportSearchScrollScanAction.java:199) at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteScan(SearchServiceTransportAction.java:377) at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.executePhase(TransportSearchScrollScanAction.java:184) at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.access$700(TransportSearchScrollScanAction.java:80) at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction$2.run(TransportSearchScrollScanAction.java:157) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IndexOutOfBoundsException: index (0) must be less than size (0) at org.elasticsearch.common.base.Preconditions.checkElementIndex(Preconditions.java:301) at org.elasticsearch.common.base.Preconditions.checkElementIndex(Preconditions.java:280) at org.elasticsearch.common.collect.Iterables.get(Iterables.java:649) at org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:259) at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.innerFinishHim(TransportSearchScrollScanAction.java:232) at org.elasticsearch.action.search.type.TransportSearchScrollScanAction$AsyncAction.finishHim(TransportSearchScrollScanAction.java:207) ... 9 more ```

-- jim

On Wed, Jun 15, 2011 at 11:38 PM, Clinton Gormley
clinton@iannounce.co.ukwrote:

On Wed, 2011-06-15 at 19:53 -0400, James Cook wrote:

Is it necessary to pass the scrollId timeout value in each subsequent
search scroll request?

The scroll timeout and the new _scroll_id from the previous search or
scroll request.

The final request will return zero hits, which is how you know that
you're done.

clint

Clinton_Gormley · June 16, 2011, 6:50am

Hi James

It doesn't seem to work that way. At least anecdotally, I don't have
to pass the timeout value in subsequent search scroll requests.

I wish the last result did return 0 hits, but instead it is currently
(0.16.2) throwing an exception.
Scroll search always throws IndexOutOfBoundsException on last iteration · Issue #1008 · elastic/elasticsearch · GitHub

Is the fact that you're not passing the timeout not the reason that you
are seeing the exception?

I use scrolling as I described, and it works without any errors. Note:
if you don't pass the timeout, you may still see a few successful scroll
results, but it won't last

clint

James_Cook · June 16, 2011, 7:15am

I attached a simple gist to the issue to recreate. If it succeeds for you,
perhaps there is a config parameter which is different between our set ups
which has some impact on the results?

So, the timeout value needs to be constantly supplied on each successive
search. I can't think of a use case where that is a helpful feature.
Couldn't each time a scroll_id is referenced, it updates its TTL with the
original value. I suppose passing the timeout each time only makes sense if:

a) The TTL needs to change while retrieving pages of results, or
b) ES doesn't have a way of knowing what the original TTL was.

Either way, it is a simple workaround even though it is a bit strange.

-- jim

On Thu, Jun 16, 2011 at 2:50 AM, Clinton Gormley clinton@iannounce.co.ukwrote:

Hi James

It doesn't seem to work that way. At least anecdotally, I don't have
to pass the timeout value in subsequent search scroll requests.

I wish the last result did return 0 hits, but instead it is currently
(0.16.2) throwing an exception.
Scroll search always throws IndexOutOfBoundsException on last iteration · Issue #1008 · elastic/elasticsearch · GitHub

Is the fact that you're not passing the timeout not the reason that you
are seeing the exception?

I use scrolling as I described, and it works without any errors. Note:
if you don't pass the timeout, you may still see a few successful scroll
results, but it won't last

clint

Clinton_Gormley · June 16, 2011, 7:31am

Hi James

On Thu, 2011-06-16 at 03:15 -0400, James Cook wrote:

I attached a simple gist to the issue to recreate. If it succeeds for
you, perhaps there is a config parameter which is different between
our set ups which has some impact on the results?

I've got no special config

Here's an example of a scrolled search which works for me on 0.16.2:

gist.github.com

https://gist.github.com/clintongormley/1028836

gistfile1.txt

# [Thu Jun 16 09:28:10 2011] Protocol: http, Server: 192.168.5.103:9200
curl -XPUT 'http://127.0.0.1:9200/test/?pretty=1' 

# [Thu Jun 16 09:28:11 2011] Response:
# {
#    "ok" : true,
#    "acknowledged" : true
# }

# [Thu Jun 16 09:29:10 2011] Protocol: http, Server: 192.168.5.103:9200

This file has been truncated. show original

clint

James_Cook · June 16, 2011, 1:44pm

Thanks clinton. Were you able to duplicate my recreation and the exception?
On Jun 16, 2011 3:31 AM, "Clinton Gormley" clinton@iannounce.co.uk wrote:

Hi James

On Thu, 2011-06-16 at 03:15 -0400, James Cook wrote:

I attached a simple gist to the issue to recreate. If it succeeds for
you, perhaps there is a config parameter which is different between
our set ups which has some impact on the results?

I've got no special config

Here's an example of a scrolled search which works for me on 0.16.2:

Scrolled search · GitHub

clint

Clinton_Gormley · June 16, 2011, 2:21pm

Hi James

On Thu, 2011-06-16 at 09:44 -0400, James Cook wrote:

Thanks clinton. Were you able to duplicate my recreation and the
exception?

I don't use the Java API I'm afraid (or Java for that matter), so no

but if you look through the curl script that i linked to, you can check
that you're doing the same steps that i did, and if there is a
difference, then that's probably where the issue is.

if there isn't, well then it may be a bug

clint

Scrolled search · GitHub

clint

James_Cook · June 16, 2011, 3:30pm

It's not a Java recreation. It's Curl.

Check out: Scroll search always throws IndexOutOfBoundsException on last iteration · Issue #1008 · elastic/elasticsearch · GitHub

It's very simple:

curl -XPOST 'http://localhost:9200/twitter/tweet/1' -d '{ "user": "kimchy"
}'

curl -XGET 'localhost:9200/_search?search_type=scan&scroll=5m&pretty=true'
-d '{ "query" : { "term": {"user":"kimchy"} } }'

get scrollID

curl -GET 'localhost:9200/_search/scroll?scroll=5m&pretty=true' -d
''

returns 1 hit

curl -GET 'localhost:9200/_search/scroll?scroll=5m&pretty=true' -d
''

throws exception

On Thu, Jun 16, 2011 at 10:21 AM, Clinton Gormley
clinton@iannounce.co.ukwrote:

Hi James

On Thu, 2011-06-16 at 09:44 -0400, James Cook wrote:

Thanks clinton. Were you able to duplicate my recreation and the
exception?

I don't use the Java API I'm afraid (or Java for that matter), so no

but if you look through the curl script that i linked to, you can check
that you're doing the same steps that i did, and if there is a
difference, then that's probably where the issue is.

if there isn't, well then it may be a bug

clint

Scrolled search · GitHub

clint

Clinton_Gormley · June 16, 2011, 3:53pm

Sorry - I'm flu ridden - I missed that:

get scrollID

curl -GET 'localhost:9200/_search/scroll?scroll=5m&pretty=true' -d
''

returns 1 hit

curl -GET 'localhost:9200/_search/scroll?scroll=5m&pretty=true' -d
''

Which scroll ID are you passing to the last statement? The scroll ID
from the search? Or the scroll ID from the previous scroll request?

It should be the latter

clint

kimchy · June 16, 2011, 4:22pm

James I will check you case. Providing the scroll parameter (with the timeout) means that you want to continue scrolling the request. Not passing it means that you don't. When scrolling, you need to make sure that you pass the scroll id you got from the previous response to the next scroll request.

On Thursday, June 16, 2011 at 6:53 PM, Clinton Gormley wrote:

Sorry - I'm flu ridden - I missed that:

get scrollID

curl -GET 'localhost:9200/_search/scroll?scroll=5m&pretty=true' -d
''

returns 1 hit

curl -GET 'localhost:9200/_search/scroll?scroll=5m&pretty=true' -d
''
Which scroll ID are you passing to the last statement? The scroll ID
from the search? Or the scroll ID from the previous scroll request?

It should be the latter

clint

James_Cook · June 16, 2011, 5:47pm

Thanks Shay and Clinton. That must be the problem. I have been using the
scroll_id from the very first "setup" request to make subsequent requests.
I'll add a comment to the issue if this is the case.

--- jim

James I will check you case. Providing the scroll parameter (with the

timeout) means that you want to continue scrolling the request. Not passing
it means that you don't. When scrolling, you need to make sure that you pass
the scroll id you got from the previous response to the next scroll request.

On Thursday, June 16, 2011 at 6:53 PM, Clinton Gormley wrote:

Sorry - I'm flu ridden - I missed that:

get scrollID

curl -GET 'localhost:9200/_search/scroll?scroll=5m&pretty=true' -d
''

returns 1 hit

curl -GET 'localhost:9200/_search/scroll?scroll=5m&pretty=true' -d
''

Which scroll ID are you passing to the last statement? The scroll ID
from the search? Or the scroll ID from the previous scroll request?

It should be the latter

clint