# Scrolling with auto-scaling? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 30, 2013, 4:21pm UTC](https://discuss.elastic.co/t/scrolling-with-auto-scaling/11752 "2013-04-30T16:21:12Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [April 30, 2013, 4:21pm UTC](https://discuss.elastic.co/t/scrolling-with-auto-scaling/11752/1 "2013-04-30T16:21:12Z")
Hi, I need to update my mapping and re-index, so I have a script to scrape the index using scrolling. My elasticsearch instances are deployed via AWS elastic beanstalk, and I have auto-scaling set up to add more instances when request volume is above a certain threshold. I've noticed that when auto-scaling takes effect, I start seeing exceptions when using the scroll id. I'm trying to work out if this is a problem in my setup, or a limitation of scrolling. my question is, should I be able to add/remove nodes in a cluster successfully while a scroll is open? --paul -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [April 30, 2013, 7:40pm UTC](https://discuss.elastic.co/t/scrolling-with-auto-scaling/11752/2 "2013-04-30T19:40:29Z")
Hi Paul The likeliest issue is that you are not using the new scroll\_id that is returned each time you pull another batch of results via the scroll request. Of course, you haven't mentioned what errors you see, so I may be barking up the wrong tree clint On Tue, Apr 30, 2013 at 6:21 PM, Paul Sanwald [paul@redowlanalytics.com](mailto:paul@redowlanalytics.com)wrote: > Hi, > I need to update my mapping and re-index, so I have a script to scrape > the index using scrolling. My elasticsearch instances are deployed via AWS > elastic beanstalk, and I have auto-scaling set up to add more instances > when request volume is above a certain threshold. > > I've noticed that when auto-scaling takes effect, I start seeing > exceptions when using the scroll id. I'm trying to work out if this is a > problem in my setup, or a limitation of scrolling. my question is, should I > be able to add/remove nodes in a cluster successfully while a scroll is > open? > > --paul > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [April 30, 2013, 7:45pm UTC](https://discuss.elastic.co/t/scrolling-with-auto-scaling/11752/3 "2013-04-30T19:45:40Z")
If you are getting SearchContextMissingException, you are most likely running into scroll limitations. Unlike all other requests, scroll is stateful and this state is kept on a certain shards. If shards are getting relocated to other nodes while scroll is running on these shards, your scroll request might partially fail. On Tuesday, April 30, 2013 12:21:12 PM UTC-4, Paul Sanwald wrote: > Hi, > I need to update my mapping and re-index, so I have a script to scrape > the index using scrolling. My elasticsearch instances are deployed via AWS > elastic beanstalk, and I have auto-scaling set up to add more instances > when request volume is above a certain threshold. > > I've noticed that when auto-scaling takes effect, I start seeing > exceptions when using the scroll id. I'm trying to work out if this is a > problem in my setup, or a limitation of scrolling. my question is, should I > be able to add/remove nodes in a cluster successfully while a scroll is > open? > > --paul -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [May 1, 2013, 1:17pm UTC](https://discuss.elastic.co/t/scrolling-with-auto-scaling/11752/4 "2013-05-01T13:17:53Z")
Clinton, you are right, this is definitely a problem. I've been using the original scroll id the whole time, I'll fix this. --paul On Tuesday, April 30, 2013 3:40:29 PM UTC-4, Clinton Gormley wrote: > Hi Paul > > The likeliest issue is that you are not using the new scroll\_id that is > returned each time you pull another batch of results via the scroll request. > > Of course, you haven't mentioned what errors you see, so I may be barking > up the wrong tree > > clint > > On Tue, Apr 30, 2013 at 6:21 PM, Paul Sanwald \<[pa...@redowlanalytics.com](mailto:pa...@redowlanalytics.com)\ > > > wrote: > > > Hi, > > I need to update my mapping and re-index, so I have a script to scrape > > the index using scrolling. My elasticsearch instances are deployed via AWS > > elastic beanstalk, and I have auto-scaling set up to add more instances > > when request volume is above a certain threshold. > > > > I've noticed that when auto-scaling takes effect, I start seeing > > exceptions when using the scroll id. I'm trying to work out if this is a > > problem in my setup, or a limitation of scrolling. my question is, should I > > be able to add/remove nodes in a cluster successfully while a scroll is > > open? > > > > --paul > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [May 1, 2013, 1:27pm UTC](https://discuss.elastic.co/t/scrolling-with-auto-scaling/11752/5 "2013-05-01T13:27:56Z")
Thanks Igor, this is the exception I'm seeing (which I failed to mention in my initial post). I am also not ever updating the scroll\_id I'm using, which is a bug in my code. it's possible I'm getting a different scroll id back and not using it, which could cause the state not to transfer correctly. --paul On Tuesday, April 30, 2013 3:45:40 PM UTC-4, Igor Motov wrote: > If you are getting SearchContextMissingException, you are most likely > running into scroll limitations. Unlike all other requests, scroll is > stateful and this state is kept on a certain shards. If shards are getting > relocated to other nodes while scroll is running on these shards, your > scroll request might partially fail. > > On Tuesday, April 30, 2013 12:21:12 PM UTC-4, Paul Sanwald wrote: > > > Hi, > > I need to update my mapping and re-index, so I have a script to scrape > > the index using scrolling. My elasticsearch instances are deployed via AWS > > elastic beanstalk, and I have auto-scaling set up to add more instances > > when request volume is above a certain threshold. > > > > I've noticed that when auto-scaling takes effect, I start seeing > > exceptions when using the scroll id. I'm trying to work out if this is a > > problem in my setup, or a limitation of scrolling. my question is, should I > > be able to add/remove nodes in a cluster successfully while a scroll is > > open? > > > > --paul -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:38am UTC](https://discuss.elastic.co/t/scrolling-with-auto-scaling/11752/6 "2017-07-06T02:38:50Z")
Closed on July 6, 2017, 2:38am UTC