Issue with forced shard allocation awareness

Elastic Stack Elasticsearch
1

I recently set up a small test cluster on docker in order to show how forced shard allocation awareness works and was surprised to find that my configuration is not working as expected. As this is old and trusted functionality that I have used repeatedly over the years I suspect I have done something wrong, but I can not spot it so thought it might be useful to see if anyone here have any ideas.

I have set up a small 3-node cluster using docker compose based on the sample provided in the docs. The compose file is available here. Did run it with Elasticsearch 8.13.4.

For 2 of the nodes have followed the example in the docs and set the following:

- node.attr.zone=A
- cluster.routing.allocation.awareness.attributes=zone
- cluster.routing.allocation.awareness.force.zone.values=A,B

The third node in the cluster instead has the following setting:

- node.attr.zone=B
- cluster.routing.allocation.awareness.attributes=zone
- cluster.routing.allocation.awareness.force.zone.values=A,B

I then logged into the dev console in Kibana and ran these commands. I first tried to see if the custom settings were visible through the _cluster/settings API, which they were not.

The _cat/nodeattrs API however showed that the zone attribute was correctly set across the cluster.

I then created 4 test indices with the default settings (1 primary and 1 replica shard) and verified that these were allocated using shard allocation awareness. The only node in zone B has a copy of each testindex shard, which is what I would expect.

To show that forced shard allocation awareness prevents allocation of multiple shard copies within the same zone I then increased the number of replicas to 2 for one of the test indices. I was expecting to see this new shard be unallocated as the only remaining node where it can be allocated is in zone A, which already has a copy. To my surprise all 3 shards were allocated and the cluster allocation explain API just stated that it is indeed allocated and does not mention awareness at all.

2

This isn't what forced awareness does - this feature exists to prevent one zone from being overloaded if another zone goes missing, but with 3 shard copies and 2 zones that means that no zone may have more than ceil(3/2) = 2 shard copies.

If you pass the ?include_yes_decisions query parameter then you will get all the deciders' outputs, including one for awareness. It won't be very useful tho, it'll just say node meets all awareness attribute requirements.

3

I think it would be useful to have that calculation specified in the documentation as that clarifies the behaviour. I deduce I am basically suffering from the mistake of shrinking the example down to 3 nodes with 2 zones instead of deploying the full 3 zones with 2+ nodes I would usually do. If I had 3 zones with 2 nodes each with 2 replica shards the maximum number of shards per zone would be 1 and I would in that case see the behaviour I am used to seeing.

4

I know what you mean, but I believe the exact formula is deliberately omitted from the reference docs so that there is freedom to change the implementation in the future. I could see some scope to improve the allocation explain message perhaps.