Elasticsearch 7.17.10 indexing bottleneck on i3.2xlarge and d3.2xlarge nodes in EKS

[Chris_Austin (Christopher Austin)]

@Christian_Dahlqvist

Are there several shards from the same hot index on these nodes or just shards from different hot indices? If it is the former you can use the index.routing.allocation.total_shards_per_node index setting to limit the number of shards from a specific index that can reside on a single node. This counts both primaries and replicas you need to set it at a level so that all of these can be allocated. You may also need to remove this setting when you move indices to the warm tier.

Wrote a quick script to check this. Most indices only have a single shard per node, but one (the busiest) has 2 shards on 3 nodes. hot-0,9,10 each have one primary and one replica (of different shards in that index). I'll reduce the primary shard count for that index to ensure only one per node. The affected nodes aren't in the top 2 for cpu, but it's tidier this way and avoids concentrating the load.