Best practice for shard rebalancing when adding nodes

Elastic Stack Elasticsearch
1

I added two new data nodes into a 4-node cluster. But I noticed that after I added the two new nodes, the new index (5 shards) is only allocated to the two new nodes. As meanwhile, it is balancing the old indices as well from other nodes to the two new nodes, and this causes the very high load for the two new nodes. But after all nodes are balanced( took hours), the shards are balanced to all nodes and the load is balanced across all data nodes. Can someone suggest what will be the best practice to avoid this?

I want to achieve the following results

  • When new nodes are added, index ingestion is happening for all nodes, not just the newly added nodes. This is not good for the load and causes index ingestions to fail.
  • In the background, it can balance the shards, but not with a high priority.

Thanks

2

There is an open issue about this problem: #17213. We are still working on ideas for a solution, but in the meantime you can use index.routing.allocation.total_shards_per_node to spread out today's shards or else exclude new shards from the new nodes. Then when the cluster has finished rebalancing you can remove these countermeasures.

3

Thanks for sharing the information.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.