We have build a new cluster with rack and zone awareness, and in our testing we have found shards are being distributed on rack properly, but we have issues with distributing shards based on zones
Settings in elasticsearch.yml
node.attr.rack_id: az2 node.attr.zone: cap2
PUT /_cluster/settings { "transient": { "cluster.routing.allocation.awareness.attributes": "rack_id,zone" } }
And the behavior is
Created index with 2 replicas and 5 shards, and here is how it distributed .
Index1 created primary and replica shard
I assume you want to make sure shards are distributed across different rack_id in order to make sure a failing rack can not affect all replicas of a shard. For this shard allocations awareness is the correct mechanism to use.
For the zone parameter I suspect you however want all shard in a single zone, so here you should instead use shard allocation filtering rather than awareness.
Thank you for quick reply, I have read shard allocation filtering here is what we are looking for
Some mechanism to make sure every shard is in different rack and then with in the rack it should be in different zone (2 shards shouldn't be on the same zone) , this helps when one zone is down we still have a copy of that in different zone.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.