we have cluster with 100 node and our index name are base on date. with elastic default options, the situation is like this:
the shard of index that begin with 2017-June are in first 10 node
the shard of index that begin with 2017-May are in second 10 node
the shard of index that begin with 2017-April are in third 10 node
and so on for every month.
the problem is workload on the 10 node that have last month index are very high and we want to distribute the shards and work load.
I think elastic have option for this situation: cluster.routing.allocation.balance.index cluster.routing.allocation.balance.shard with it get a float number.
Am I correct with this? how much i should raise the float number?
any explanation will be appreciate
thanks.
How did you end up with that kind of distribution in the first place? I would expect Elasticsearch to spread it out quite evenly by default.
I don't know. as I said, all setting are default and we didn't change any specific options.
How many shards do you have per node? Are you using daily indices?
Can you show the configuration for one of the nodes as well as the index settings for an index on that node?
yes. we use daily indices.
at least 100 shards per node.
That sounds like a very large number of potentially small shards, which can be very inefficient. Please read this blog post for some practical guidelines around sharding and shard sizes.
no. the shards at least have 50 GB data and they are not small
thank you for responding but the problem is about how to distribute last month shards between 50% of nodes.
If your shards are at least 50GB in size and you have over 1000 shards per node, that means over 50TB of data per node, is that correct? Are all these indices open? Can you perhaps provide the output of the cluster stats API?
sorry. I did a mistake. 100 shards per node.
yes we have 5 TB per node.
I cant send you cluster stats now.
Then it sounds like you are following best practices. 
can you describe cluster.routing.allocation.balance.shard option?
How many shards do you have per index? How large portion of indices related to June 2017 are located on this set of 10 nodes? Is there more than 1 shard for any particular index on any one node?
between 5 to 10 shards according to indices.
no. there is not more than 1 shard of a index on any node.
Then those settings may not help much as they look to distribute shards of individual indices across nodes. You may need to manually reroute indices.
1 Like
How many indices per month are being created?
Wild guess: are you perhaps deleting some "old" indices just before creating the new ones each month? If so, Elasticsearch may be preferring to put the new shards where the old ones have left gaps, and if the old indices were all concentrated on 10 nodes then the new ones will be too.
If it is this, then creating the new indices without first deleting the old ones should cause them to be allocated more evenly.
1 Like
thanks for responds.
we don't delete any index.