# Resuming scroll request after intermediate failure
**URL:** https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178
**Category:** Elasticsearch
**Created:** [May 25, 2017, 7:50pm UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178 "2017-05-25T19:50:38Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [May 25, 2017, 7:50pm UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/1 "2017-05-25T19:50:38Z")
If we are scrolling through a large index (say 50M) doc and one of the requests after scrolling through 99% of the docs fail, it looks like we have to start all over again, which is damn expensive and feels inefficient.
I hope I am missing something. I wanted to check if there is a way to resume scroll request. Using the last scroll\_id does not return the same set of documents.
Please suggest how to handle this failure scenario.
---
### Author:  [@thiago](https://discuss.elastic.co/u/thiago)
#### Post date: [May 29, 2017, 2:20am UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/2 "2017-05-29T02:20:56Z")
Hello,
What kinda failure are you referring to? Can you please post the error message?
Cheers
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [May 30, 2017, 4:23am UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/3 "2017-05-30T04:23:07Z")
This is just a general design question so that we can handle error messaging appropriately. Failing at 99% and restarting didn't make much sense to me (there wasn't any failure since we haven't used it yet), so wanted to double check from experts.
---
### Author:  [@thiago](https://discuss.elastic.co/u/thiago)
#### Post date: [May 30, 2017, 12:12pm UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/4 "2017-05-30T12:12:16Z")
A scroll context is stateful and is bound to the nodes where the primary scrolling shards are assigned and there is no high availability guarantees, which means that if one of those node fails, since there is no scroll context replication, then the scroll will fail. Another reason for it to fail is if index is closed and/or deleted.
Besides the reasons described above, I don't recall any other one that could cause a scroll context to fail (besides it simply timing out). That's why asked for you to post the error.
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 9, 2017, 4:09pm UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/5 "2017-06-09T16:09:30Z")
Sorry for the late reply, was out on vacation.
Thanks for the details about the scroll request. A error can happen because of network issue (the major root cause of majority of the live site issues on production). The error could be on client library as well (not necessarily elasticsearch cluster/node). To have the capability to continue using retry mechanism would be super useful in that case.
Again, I don't have any error messages yet, but it makes us nervous to use this feature in production because we don't have a good answer (there is a workaround, albeit an expensive one) to handle failure other than restarting all over again.
---
### Author:  [@thiago](https://discuss.elastic.co/u/thiago)
#### Post date: [June 9, 2017, 4:35pm UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/6 "2017-06-09T16:35:38Z")
The scroll context will stay open for a specified amount of time. So in case of a network error, you can repeat the request for the same scroll id. But keep in mind that, depending on what type of network error, it could be the case that the request actually made it and Elasticsearch processed it, so if you could not get the response (because of network error) then the next request will be the next page in scroll context.
Also, you mentioned using scrolled request in production, are you referring to use these type of requests for the scale of user initiated searches?
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 9, 2017, 6:28pm UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/7 "2017-06-09T18:28:21Z")
I could retry, but there is no way to know if the request was successful on server side. This means I could miss documents in the scroll request.
We use scrolling for exporting complete index out of ES in production. That's an option that clients control.
---
### Author:  [@thiago](https://discuss.elastic.co/u/thiago)
#### Post date: [June 9, 2017, 6:36pm UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/8 "2017-06-09T18:36:03Z")
> [@animageofmine](#):
>
> I could retry, but there is no way to know if the request was successful on server side. This means I could miss documents in the scroll request.
Indeed this is a problem that can actually happen. I suggest that you explore other options:
1. Maybe you could use [Search After](https://www.elastic.co/guide/en/elasticsearch/reference/5.x/search-request-search-after.html). It works differently from scroll context, but also with caveats on it's own, but maybe it is suitable for your case.
2. Is this an on-prem cluster that you control? If so, you can deploy Spark and es-hadoop to export the documents directly from the data nodes.
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 10, 2017, 1:16am UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/9 "2017-06-10T01:16:58Z")
> [@thiago](#):
>
> Is this an on-prem cluster that you control? If so, you can deploy Spark and es-hadoop to export the documents directly from the data nodes.
from what I understand, search\_after is expensive for exports. Would you mind elaborating a bit more about the spark and es-hadoop for exporting data?
---
### Author:  [@thiago](https://discuss.elastic.co/u/thiago)
#### Post date: [June 10, 2017, 1:27am UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/10 "2017-06-10T01:27:24Z")
You would create a Spark job that reads document from Elasticsearch (this integration is provided by es-hadoop) and writes somewhere else. You could even overlay the Spark cluster with Elasticsearch cluster for job+data collocation, this would make job more resilient and less error prone.
Check [https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 8, 2017, 1:27am UTC](https://discuss.elastic.co/t/resuming-scroll-request-after-intermediate-failure/87178/11 "2017-07-08T01:27:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.