Balancing shards equally per nodes

Elastic Stack Elasticsearch
1

Hy,

I have a ES cluster running 1.6.0, consisting of 10 nodes.

2 of them are new nodes in this cluster, with small number of shards( hundreds).
The other 8 nodes have thousands of shards.

The problem is that when a new index (I create indexes with 10 shards and 0 replicas), all shards from the newly index goes to those 2 nodes and I want to distribute equally in the cluster. (This means one shard/node).

I tried without luck the following settings:
"persistent": {"cluster": {"routing": {"allocation": {"balance": {"index": "0.7","shard": "0.3"}}}}}
"persistent": {"cluster": {"routing": {"allocation": {"balance": {"index": "0.8","shard": "0.2"}}}}}
"persistent": {"cluster": {"routing": {"allocation": {"balance": {"index": "2","shard": "0.1"}}}}}

How can I do this?

Thank you,

2

Have a look at https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-total-shards.html

Also, you should be better off with higher numbers for the shard setting. It sounds like the cluster hasn't fully balanced yet after the new nodes. Is that so?

Also, hundreds of shards is a lot. Like, thousands of shards is ok, but has significant overhead.

3

Thanks Nik for your reply.

index.routing.allocation.total_shards_per_node can be set on each index or is a cluster variable?

Right now, it seems that with the following settings:

"persistent": {"cluster": {"routing": {"allocation": {"balance": {"index": "10","shard": "0.1"}}}}},

the shards of newly created indexes are distributed evenly.

1 Like