I'm looking for a way to limit the number of shards per nodes on a group of index. Is it possible, may be by using index patterns ?
My situation is :
- I have a set of 14 indexes with 12 shards for each
- We have 10 nodes
- I need those 14 indexes to be spread as flat as possible on the nodes, meaning 16-17 shards by nodes. However we have many other indexes in the stack, and elasticsearch sometimes changes the shards repartition for my 14 indexes (after a node falling, or because we lower the load on those 14 indexes, etc.).
- Therefore I would love to be able to set something like
Would you have an idea to share ?
Thank you !
total_shards_per_node is an index level setting, you cannot assign a specific value to a group of indices. You have to deal with individual index and set it to 2 (for instance) in order to avoid hotspots.
Be aware that this is a hard limit to the number of shards (primary+replica) of these indices that each data node will accept.
Out of curiosity, why 12 shards per index ? are they only primaries or does it include replica ?
Hello Vincent, thank you for your answer !
The thing is that the limit of 2 / index allows hotspots of 28 shards and only that destroys my performances.
I don't quite understand why ES rebalancing choses to create hotspot while my indexes are (normally) solicited in a homogeneous way. And a proof for that is that as soon as a hotspot appears, the node's latency is 3 to 4 times greater than the other nodes latencies.
We have only primaries for those indexes and 12 shards makes each shard being 50Go and this repartitions seems optimal for our performances
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.