Re-balancing shard allocation

(Hamid Bayat) #1

we have cluster with 100 node and our index name are base on date. with elastic default options, the situation is like this:

the shard of index that begin with 2017-June are in first 10 node
the shard of index that begin with 2017-May are in second 10 node
the shard of index that begin with 2017-April are in third 10 node
and so on for every month.

the problem is workload on the 10 node that have last month index are very high and we want to distribute the shards and work load.

I think elastic have option for this situation: cluster.routing.allocation.balance.index cluster.routing.allocation.balance.shard with it get a float number.

Am I correct with this? how much i should raise the float number?

any explanation will be appreciate

(Christian Dahlqvist) #2

How did you end up with that kind of distribution in the first place? I would expect Elasticsearch to spread it out quite evenly by default.

(Hamid Bayat) #3

I don't know. as I said, all setting are default and we didn't change any specific options.

(Christian Dahlqvist) #4

How many shards do you have per node? Are you using daily indices?

Can you show the configuration for one of the nodes as well as the index settings for an index on that node?

(Hamid Bayat) #5

yes. we use daily indices.
at least 100 shards per node.

(Christian Dahlqvist) #6

That sounds like a very large number of potentially small shards, which can be very inefficient. Please read this blog post for some practical guidelines around sharding and shard sizes.

(Hamid Bayat) #7

no. the shards at least have 50 GB data and they are not small

thank you for responding but the problem is about how to distribute last month shards between 50% of nodes.

(Christian Dahlqvist) #8

If your shards are at least 50GB in size and you have over 1000 shards per node, that means over 50TB of data per node, is that correct? Are all these indices open? Can you perhaps provide the output of the cluster stats API?

(Hamid Bayat) #9

sorry. I did a mistake. 100 shards per node.
yes we have 5 TB per node.

(Hamid Bayat) #10

I cant send you cluster stats now.

(Christian Dahlqvist) #11

Then it sounds like you are following best practices. :slight_smile:

(Hamid Bayat) #12

can you describe cluster.routing.allocation.balance.shard option?

(Christian Dahlqvist) #13

How many shards do you have per index? How large portion of indices related to June 2017 are located on this set of 10 nodes? Is there more than 1 shard for any particular index on any one node?

(Hamid Bayat) #14

between 5 to 10 shards according to indices.
no. there is not more than 1 shard of a index on any node.

(Christian Dahlqvist) #15

Then those settings may not help much as they look to distribute shards of individual indices across nodes. You may need to manually reroute indices.

(Hamid Bayat) #16

thanks for the answers.

(David Turner) #17

How many indices per month are being created?

(Hamid Bayat) #19

60 indices.

(David Turner) #20

Wild guess: are you perhaps deleting some "old" indices just before creating the new ones each month? If so, Elasticsearch may be preferring to put the new shards where the old ones have left gaps, and if the old indices were all concentrated on 10 nodes then the new ones will be too.

If it is this, then creating the new indices without first deleting the old ones should cause them to be allocated more evenly.

(Hamid Bayat) #21

thanks for responds.
we don't delete any index.