Elasticsearch [6.8] Shard distribution based on traffic

Elastic Stack Elasticsearch
1

Hi Folks,
We are seeing scenarios where the cluster is allocating 6 -10 shards(primaries & replicas) of an index with shard sizes around 50GB on a single node of the cluster even though all nodes in the cluster are of the same configuration. This behavior is causing issues when we have a high throughput of writes on that index coming in & the load is on that node which is housing these large shards on it & at times results in thread_pools maxing out & we encounter rejections

Want to understand what is the logic behind this behavior & is there any allocation algo that distributes the shards based on traffic ? so that the shard gets allocated to avoid the above mentioned scenario.

2

Allocation of shards is largely driven by shard count. You can ensure a more even distribution by setting index.routing.allocation.total_shards_per_node in your index template. Make sure to set this so that the cluster has the ability to relocate shards in case of node failure.

Also be aware that version 6.8 is EOL, so I would recommend you upgrade to a newer version.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.