Hi,
I need to update my mapping and re-index, so I have a script to scrape
the index using scrolling. My elasticsearch instances are deployed via AWS
elastic beanstalk, and I have auto-scaling set up to add more instances
when request volume is above a certain threshold.
I've noticed that when auto-scaling takes effect, I start seeing exceptions
when using the scroll id. I'm trying to work out if this is a problem in my
setup, or a limitation of scrolling. my question is, should I be able to
add/remove nodes in a cluster successfully while a scroll is open?
--paul
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Hi Paul
The likeliest issue is that you are not using the new scroll_id that is
returned each time you pull another batch of results via the scroll request.
Of course, you haven't mentioned what errors you see, so I may be barking
up the wrong tree
clint
On Tue, Apr 30, 2013 at 6:21 PM, Paul Sanwald paul@redowlanalytics.comwrote:
Hi,
I need to update my mapping and re-index, so I have a script to scrape
the index using scrolling. My elasticsearch instances are deployed via AWS
elastic beanstalk, and I have auto-scaling set up to add more instances
when request volume is above a certain threshold.
I've noticed that when auto-scaling takes effect, I start seeing
exceptions when using the scroll id. I'm trying to work out if this is a
problem in my setup, or a limitation of scrolling. my question is, should I
be able to add/remove nodes in a cluster successfully while a scroll is
open?
--paul
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
If you are getting SearchContextMissingException, you are most likely
running into scroll limitations. Unlike all other requests, scroll is
stateful and this state is kept on a certain shards. If shards are getting
relocated to other nodes while scroll is running on these shards, your
scroll request might partially fail.
On Tuesday, April 30, 2013 12:21:12 PM UTC-4, Paul Sanwald wrote:
Hi,
I need to update my mapping and re-index, so I have a script to scrape
the index using scrolling. My elasticsearch instances are deployed via AWS
elastic beanstalk, and I have auto-scaling set up to add more instances
when request volume is above a certain threshold.
I've noticed that when auto-scaling takes effect, I start seeing
exceptions when using the scroll id. I'm trying to work out if this is a
problem in my setup, or a limitation of scrolling. my question is, should I
be able to add/remove nodes in a cluster successfully while a scroll is
open?
--paul
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Clinton, you are right, this is definitely a problem. I've been using the
original scroll id the whole time, I'll fix this.
--paul
On Tuesday, April 30, 2013 3:40:29 PM UTC-4, Clinton Gormley wrote:
Hi Paul
The likeliest issue is that you are not using the new scroll_id that is
returned each time you pull another batch of results via the scroll request.
Of course, you haven't mentioned what errors you see, so I may be barking
up the wrong tree
clint
On Tue, Apr 30, 2013 at 6:21 PM, Paul Sanwald <pa...@redowlanalytics.com<javascript:>
wrote:
Hi,
I need to update my mapping and re-index, so I have a script to scrape
the index using scrolling. My elasticsearch instances are deployed via AWS
elastic beanstalk, and I have auto-scaling set up to add more instances
when request volume is above a certain threshold.
I've noticed that when auto-scaling takes effect, I start seeing
exceptions when using the scroll id. I'm trying to work out if this is a
problem in my setup, or a limitation of scrolling. my question is, should I
be able to add/remove nodes in a cluster successfully while a scroll is
open?
--paul
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Thanks Igor, this is the exception I'm seeing (which I failed to mention in
my initial post). I am also not ever updating the scroll_id I'm using,
which is a bug in my code. it's possible I'm getting a different scroll id
back and not using it, which could cause the state not to transfer
correctly.
--paul
On Tuesday, April 30, 2013 3:45:40 PM UTC-4, Igor Motov wrote:
If you are getting SearchContextMissingException, you are most likely
running into scroll limitations. Unlike all other requests, scroll is
stateful and this state is kept on a certain shards. If shards are getting
relocated to other nodes while scroll is running on these shards, your
scroll request might partially fail.
On Tuesday, April 30, 2013 12:21:12 PM UTC-4, Paul Sanwald wrote:
Hi,
I need to update my mapping and re-index, so I have a script to scrape
the index using scrolling. My elasticsearch instances are deployed via AWS
elastic beanstalk, and I have auto-scaling set up to add more instances
when request volume is above a certain threshold.
I've noticed that when auto-scaling takes effect, I start seeing
exceptions when using the scroll id. I'm trying to work out if this is a
problem in my setup, or a limitation of scrolling. my question is, should I
be able to add/remove nodes in a cluster successfully while a scroll is
open?
--paul
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.