Shard relocation progress

Elastic Stack Elasticsearch
1

What's the best way to monitor shard relocation that occurrs when one add
new nodes?

Is there a way to control the relocation and do it manually with few shards
at a time?

What are the best practices for cluster that is contantly received high
volume traffic?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWovrNe4cJvqAuvPV26zErezm0eqo9rF%2B%3DFpFk_Pzh-q0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi, comment inline.

On Thu, Dec 19, 2013 at 5:28 AM, Mohit Anchlia mohitanchlia@gmail.comwrote:

What's the best way to monitor shard relocation that occurrs when one add
new nodes?

Turn on the logging and tailing the logs and start reading, but if you want
visual, you can use elasticsearch-head

Is there a way to control the relocation and do it manually with few
shards at a time?

Yes, you should spend sometime to read on the cluster settings,

in
there, you should find the parameter you want. Try experiment those
settings in your test environment.

What are the best practices for cluster that is contantly received high
volume traffic?

throttle them?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAOT3TWovrNe4cJvqAuvPV26zErezm0eqo9rF%2B%3DFpFk_Pzh-q0Q%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itznn%2BG4D2s5SNNC3QARObz8j1gj6uuJ2numKngZukxuEA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hey,

Monitor: it is all in the cluster state. Use a plugin like head or
paramedic to monitor it. If you play around with 1.0.0 beta, check out the
Cat API, which is a neat commandline interface to find out, what happens in
your cluster.

Control: You can control relocation. You can enable/disable it, set up
rules which nodes should hold which shards or if shards should be moved
around based on disk space allocation (allocation, awareness, forced
awareness).

Practices: You can throttle the amount of concurrent relocations (per
node), you can throttle its throughput to make sure that daily business
like searching and indexing still work as in lines with your SLA.

Hope this helps is there any specific hidden concrete question? :slight_smile:

--Alex

On Wed, Dec 18, 2013 at 10:28 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

What's the best way to monitor shard relocation that occurrs when one add
new nodes?

Is there a way to control the relocation and do it manually with few
shards at a time?

What are the best practices for cluster that is contantly received high
volume traffic?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAOT3TWovrNe4cJvqAuvPV26zErezm0eqo9rF%2B%3DFpFk_Pzh-q0Q%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_kR7xyCZUMidEHdpHxOZV9hPfa2TXgZ7y5mzXM8jebFA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like
4

Thanks! that helps, I'll look at these settings.

Regarding monitoring is there a way to look at the progress.

On Thu, Dec 19, 2013 at 7:13 AM, Alexander Reelsen alr@spinscale.de wrote:

Hey,

Monitor: it is all in the cluster state. Use a plugin like head or
paramedic to monitor it. If you play around with 1.0.0 beta, check out the
Cat API, which is a neat commandline interface to find out, what happens in
your cluster.

Control: You can control relocation. You can enable/disable it, set up
rules which nodes should hold which shards or if shards should be moved
around based on disk space allocation (allocation, awareness, forced
awareness).

Practices: You can throttle the amount of concurrent relocations (per
node), you can throttle its throughput to make sure that daily business
like searching and indexing still work as in lines with your SLA.

Hope this helps is there any specific hidden concrete question? :slight_smile:

--Alex

On Wed, Dec 18, 2013 at 10:28 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:

What's the best way to monitor shard relocation that occurrs when one add
new nodes?

Is there a way to control the relocation and do it manually with few
shards at a time?

What are the best practices for cluster that is contantly received high
volume traffic?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAOT3TWovrNe4cJvqAuvPV26zErezm0eqo9rF%2B%3DFpFk_Pzh-q0Q%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_kR7xyCZUMidEHdpHxOZV9hPfa2TXgZ7y5mzXM8jebFA%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr11LH-e_Z-iNp%3D15a9Y1PJhpOBUM%3DpRf4jmPEMgtuMqg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi Mohit,

On Thursday, December 19, 2013 12:38:36 PM UTC-5, Mo wrote:

Thanks! that helps, I'll look at these settings.

Regarding monitoring is there a way to look at the progress.

Of the migration state/%/ETA for a single shard? I don't think there's
anything that will show you that....
But if you have N shards moving around and want to see "progress over time"
in the sense of being able to see the number of shards moving around
currently and at different points in time/history, you can see that in SPM.

.... like in this image, more or
less: http://elasticsearch-users.115913.n3.nabble.com/attachment/3803285/0/spm-es-shard-dance.png

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Dec 19, 2013 at 7:13 AM, Alexander Reelsen <a...@spinscale.de<javascript:>

wrote:

Hey,

Monitor: it is all in the cluster state. Use a plugin like head or
paramedic to monitor it. If you play around with 1.0.0 beta, check out the
Cat API, which is a neat commandline interface to find out, what happens in
your cluster.

Control: You can control relocation. You can enable/disable it, set up
rules which nodes should hold which shards or if shards should be moved
around based on disk space allocation (allocation, awareness, forced
awareness).

Practices: You can throttle the amount of concurrent relocations (per
node), you can throttle its throughput to make sure that daily business
like searching and indexing still work as in lines with your SLA.

Hope this helps is there any specific hidden concrete question? :slight_smile:

--Alex

On Wed, Dec 18, 2013 at 10:28 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>

wrote:

What's the best way to monitor shard relocation that occurrs when one
add new nodes?

Is there a way to control the relocation and do it manually with few
shards at a time?

What are the best practices for cluster that is contantly received high
volume traffic?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAOT3TWovrNe4cJvqAuvPV26zErezm0eqo9rF%2B%3DFpFk_Pzh-q0Q%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_kR7xyCZUMidEHdpHxOZV9hPfa2TXgZ7y5mzXM8jebFA%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/97f6a19e-7dd8-4b0d-994e-6ae0199d61c9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.