# Resume scroll-scan query? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 23, 2014, 4:47pm UTC](https://discuss.elastic.co/t/resume-scroll-scan-query/20404 "2014-10-23T16:47:02Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![roger](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@roger](https://discuss.elastic.co/u/roger)\ **Post date:** [October 23, 2014, 4:47pm UTC](https://discuss.elastic.co/t/resume-scroll-scan-query/20404/1 "2014-10-23T16:47:02Z")
I'm reindexing a ElasticSearch base with 50m docs using the scroll-scan request to retrieve all docs, but my "reindexer" program stopped at 30m Is there a way to redo the query to retrieve the left docs? Like using offset? Would the the internal order of the scan query be the same with a second request? I can assure that no new docs were indexed in the old index since the beginning of the reindexing -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/69ec11c9-774e-42df-be57-fd870d347743%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/69ec11c9-774e-42df-be57-fd870d347743%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [October 23, 2014, 7:12pm UTC](https://discuss.elastic.co/t/resume-scroll-scan-query/20404/2 "2014-10-23T19:12:06Z")
The scroll is available based on a timeout value you give it. Everytimetime you scroll you restart the countdown. You could track the last scroll id you used and try it again from there? On Thursday, 23 October 2014 12:47:02 UTC-4, Roger de Cordova Farias wrote: > I'm reindexing a Elasticsearch base with 50m docs using the scroll-scan > request to retrieve all docs, but my "reindexer" program stopped at 30m > > Is there a way to redo the query to retrieve the left docs? Like using > offset? > > Would the the internal order of the scan query be the same with a second > request? > > I can assure that no new docs were indexed in the old index since the > beginning of the reindexing -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![roger](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@roger](https://discuss.elastic.co/u/roger)\ **Post date:** [October 23, 2014, 7:43pm UTC](https://discuss.elastic.co/t/resume-scroll-scan-query/20404/3 "2014-10-23T19:43:39Z")
Hmm, I was using a small ttl, just enough to process each scroll call, but I could try using a longer time to live and resuming from the last scroll\_id in case of error That is a good idea, thanks 2014-10-23 17:12 GMT-02:00 John Smith [java.dev.mtl@gmail.com](mailto:java.dev.mtl@gmail.com): > The scroll is available based on a timeout value you give it. > Everytimetime you scroll you restart the countdown. > > You could track the last scroll id you used and try it again from there? > > On Thursday, 23 October 2014 12:47:02 UTC-4, Roger de Cordova Farias wrote: > > > I'm reindexing a Elasticsearch base with 50m docs using the scroll-scan > > request to retrieve all docs, but my "reindexer" program stopped at 30m > > > > Is there a way to redo the query to retrieve the left docs? Like using > > offset? > > > > Would the the internal order of the scan query be the same with a second > > request? > > > > I can assure that no new docs were indexed in the old index since the > > beginning of the reindexing > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/NbshHCrBHoM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/NbshHCrBHoM/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJp2532aT3fhY6axy%3DRwCG3Ukh9ivP1fmqoUs3pJa65e8oAs6A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2532aT3fhY6axy%3DRwCG3Ukh9ivP1fmqoUs3pJa65e8oAs6A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [October 23, 2014, 8:20pm UTC](https://discuss.elastic.co/t/resume-scroll-scan-query/20404/4 "2014-10-23T20:20:37Z")
Small ttl is ok (well adjusted properly for you process) because everytime you call scroll it resets the ttl. So you don't need to put a 60m scroll time. It just has to be long enough to be able to process the next scroll id. I'm curious if you can re-use the scroll id. It's not specifically mentioned in the docs but i think scroll is forward only. So not sure once you got once scroll id you can go back to it. I guess one way to find out 🙂 On Thursday, 23 October 2014 15:44:04 UTC-4, Roger de Cordova Farias wrote: > Hmm, I was using a small ttl, just enough to process each scroll call, but > I could try using a longer time to live and resuming from the last > scroll\_id in case of error > > That is a good idea, thanks > > 2014-10-23 17:12 GMT-02:00 John Smith \<[java.d...@gmail.com](mailto:java.d...@gmail.com) \\>: > > > The scroll is available based on a timeout value you give it. > > Everytimetime you scroll you restart the countdown. > > > > You could track the last scroll id you used and try it again from there? > > > > On Thursday, 23 October 2014 12:47:02 UTC-4, Roger de Cordova Farias > > wrote: > > > > > I'm reindexing a Elasticsearch base with 50m docs using the scroll-scan > > > request to retrieve all docs, but my "reindexer" program stopped at 30m > > > > > > Is there a way to redo the query to retrieve the left docs? Like using > > > offset? > > > > > > Would the the internal order of the scan query be the same with a second > > > request? > > > > > > I can assure that no new docs were indexed in the old index since the > > > beginning of the reindexing > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/NbshHCrBHoM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/NbshHCrBHoM/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ec345d9e-19b4-4d2c-985a-fbf245e31a19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec345d9e-19b4-4d2c-985a-fbf245e31a19%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![roger](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@roger](https://discuss.elastic.co/u/roger)\ **Post date:** [October 23, 2014, 8:48pm UTC](https://discuss.elastic.co/t/resume-scroll-scan-query/20404/5 "2014-10-23T20:48:31Z")
I know it resets the ttl on each scroll call, but since I don't have an automatic resuming process, I need to manually check the last scroll\_id (I will log it to a file) and restart the reindexing program using it. That is why I need a longer ttl I just tested the re-use of the scroll\_id. Looks like after the first request, the same scroll\_id is returned over and over, returning new docs. So I can't use this approach, since I will always lose the last batch after resuming the reindexing 2014-10-23 18:20 GMT-02:00 John Smith [java.dev.mtl@gmail.com](mailto:java.dev.mtl@gmail.com): > Small ttl is ok (well adjusted properly for you process) because everytime > you call scroll it resets the ttl. So you don't need to put a 60m scroll > time. It just has to be long enough to be able to process the next scroll > id. > > I'm curious if you can re-use the scroll id. It's not specifically > mentioned in the docs but i think scroll is forward only. So not sure once > you got once scroll id you can go back to it. I guess one way to find out 🙂 > > On Thursday, 23 October 2014 15:44:04 UTC-4, Roger de Cordova Farias wrote: > > > Hmm, I was using a small ttl, just enough to process each scroll call, > > but I could try using a longer time to live and resuming from the last > > scroll\_id in case of error > > > > That is a good idea, thanks > > > > 2014-10-23 17:12 GMT-02:00 John Smith [java.d...@gmail.com](mailto:java.d...@gmail.com): > > > > > The scroll is available based on a timeout value you give it. > > > Everytimetime you scroll you restart the countdown. > > > > > > You could track the last scroll id you used and try it again from there? > > > > > > On Thursday, 23 October 2014 12:47:02 UTC-4, Roger de Cordova Farias > > > wrote: > > > > > > > I'm reindexing a Elasticsearch base with 50m docs using the scroll-scan > > > > request to retrieve all docs, but my "reindexer" program stopped at 30m > > > > > > > > Is there a way to redo the query to retrieve the left docs? Like using > > > > offset? > > > > > > > > Would the the internal order of the scan query be the same with a > > > > second request? > > > > > > > > I can assure that no new docs were indexed in the old index since the > > > > beginning of the reindexing > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > topic/elasticsearch/NbshHCrBHoM/unsubscribe. > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/45b51276-daa4-4f39-b46c-017296db689a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/NbshHCrBHoM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/NbshHCrBHoM/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ec345d9e-19b4-4d2c-985a-fbf245e31a19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec345d9e-19b4-4d2c-985a-fbf245e31a19%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/ec345d9e-19b4-4d2c-985a-fbf245e31a19%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ec345d9e-19b4-4d2c-985a-fbf245e31a19%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJp2532ZuYCP6%3DxqJeUmZGowAo9dzY%2BQGZkHKbfkTyCWxODF5w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2532ZuYCP6%3DxqJeUmZGowAo9dzY%2BQGZkHKbfkTyCWxODF5w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:54am UTC](https://discuss.elastic.co/t/resume-scroll-scan-query/20404/6 "2017-07-06T00:54:05Z")