ES Shard re-distribution problem

Elastic Stack Elasticsearch
1

Hi,
I've a ES cluster of 4 nodes with 3 indices each having 10 shards and 2
replicas. After indexing some data into it and running some performance
tests, I realised 4 nodes may not be enough and added an additional node.
In principle, every index should've (uniformly) re-distributed its shards.
But, to my surprise, 2 indices have redistributed only 2 shards to this new
node, while it should be 6 shards. And the third index has not transferred
even a single shard to the new node. I dont think this is an expected
behaviour from ES.
Any pointers/suggestions?

Cheers
Nitish

2

Hi,

Shard distribution is not per index. It is based on total number of shards,
so the behavior you see may be normal if the total number of shards per
node is more or less even.
There are some configuration options that you may be able to use to have a
distribution that is more inline with your needs.

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Wed, Apr 18, 2012 at 11:28 AM, Nitish Sharma
sharmanitishdutt@gmail.comwrote:

Hi,
I've a ES cluster of 4 nodes with 3 indices each having 10 shards and 2
replicas. After indexing some data into it and running some performance
tests, I realised 4 nodes may not be enough and added an additional node.
In principle, every index should've (uniformly) re-distributed its shards.
But, to my surprise, 2 indices have redistributed only 2 shards to this new
node, while it should be 6 shards. And the third index has not transferred
even a single shard to the new node. I dont think this is an expected
behaviour from ES.
Any pointers/suggestions?

Cheers
Nitish

3

Also note, we added the ability to set the number of shards per node for an
index as an interim solution to force redistribution of shards, and we plan
to improve on the allocation in the next version.

On Wed, Apr 18, 2012 at 6:40 PM, Berkay Mollamustafaoglu
mberkay@gmail.comwrote:

Hi,

Shard distribution is not per index. It is based on total number of
shards, so the behavior you see may be normal if the total number of shards
per node is more or less even.
There are some configuration options that you may be able to use to have a
distribution that is more inline with your needs.
Elasticsearch Platform — Find real-time answers at scale | Elastic

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Wed, Apr 18, 2012 at 11:28 AM, Nitish Sharma <
sharmanitishdutt@gmail.com> wrote:

Hi,
I've a ES cluster of 4 nodes with 3 indices each having 10 shards and 2
replicas. After indexing some data into it and running some performance
tests, I realised 4 nodes may not be enough and added an additional node.
In principle, every index should've (uniformly) re-distributed its shards.
But, to my surprise, 2 indices have redistributed only 2 shards to this new
node, while it should be 6 shards. And the third index has not transferred
even a single shard to the new node. I dont think this is an expected
behaviour from ES.
Any pointers/suggestions?

Cheers
Nitish