Shard rebalancing

Elastic Stack Elasticsearch
1

What is the behavior of ES when it comes to shard sizes? Does it do
automatic shard rebalancing at any point of time? If so, is it also
controlled through an API?

How can I know if the shards are changing in the background? If I do not
add any new node or change any cluster configuration once indexing has
started, is there any pattern to this behavior? Please let me know.

Thanks,
Sandeep

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f3dee22-0c33-4446-a0dc-eaf1a314d2c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Shard size will depend entirely on how many shards you've set and how big
the index is.
Allocation of data to shards happens in a round-robin manner, so balancing
isn't needed.

What do you mean by shards changing in the background?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 4 August 2014 04:45, 'Sandeep Ramesh Khanzode' via elasticsearch <
elasticsearch@googlegroups.com> wrote:

What is the behavior of ES when it comes to shard sizes? Does it do
automatic shard rebalancing at any point of time? If so, is it also
controlled through an API?

How can I know if the shards are changing in the background? If I do not
add any new node or change any cluster configuration once indexing has
started, is there any pattern to this behavior? Please let me know.

Thanks,
Sandeep

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9f3dee22-0c33-4446-a0dc-eaf1a314d2c4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9f3dee22-0c33-4446-a0dc-eaf1a314d2c4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aCRon%2B1%2BFWKnC7tkKU5oMYyHiONGaOiTJs%3DCHhE%3DezoA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

On Sun, Aug 3, 2014 at 7:43 PM, Mark Walkom markw@campaignmonitor.com
wrote:

Shard size will depend entirely on how many shards you've set and how big
the index is.
Allocation of data to shards happens in a round-robin manner, so balancing
isn't needed.

What do you mean by shards changing in the background?

The shards aren't assigned round robin - they are assigned to minimize a
cluster balance function. The cluster balance function is a combination of
three independent functions described here:

None of them take into account shard size. I'm working on one right now
that does.

For now your best bet is to set
cluster.routing.allocation.balance.index pretty high, like 80% of the sum.
That'll make all shards of a given index are smoothed across all nodes.

The new one will try to balance the count of shards of similar size across
all the nodes. That keeps one node from filling up while the others only
have small shards, for example. It should also spread load more evenly
when the indexes are different sizes.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3ah7gtmJJkHKmi9hNncU%3D8E5%3DcdLrb2LjyrBaeOZnyiw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.