We are running a 3 node Elasticsearch 7.0.1 cluster and looking for options to handle the single-node outage scenario. Since we are using the default configuration options, when single node goes down from healthy 3-node cluster, shards (primaries and replicas) of downed-node migrate to remaining two nodes as expected (via allocation + rebalance mechanisms after “index.unassigned.node_left.delayed_timeout”).
Also, when the downed-node joins back the cluster, whole rebalancing / relocation cycle gets executed. I understand that primaries relocating to remaining nodes is something which is required (without which writes will not go through for corresponding shards).
Specific clarifications :
a. What happens to the stale replica copies when downed node joins back? I assume they get fully destroyed and re-allocation and rebalancing results in replicas getting created from scratch on node which joined. Please correct me if I am wrong
b. Are there any config options to avoid/minimize the churn due to relocation/rebalance of shards when single-node goes down and when it joins back the cluster ? We are trying to minimize or avoid above churn which can potentially impact read/write availability of shards during the period of relocation + rebalancing
Our typical index config + operational params :
Replicas : 1 per shard
Shards : 5 per index
Shards per node : ~700-800 shards
Wait-for-active shards : currently using default value of 1 (primary only). But considering value of 2 (primary+replica) for maximizing the replica-consistency
Cluster-network : Local to datacenter
Some basic constraints of deployment :
- Adding additional data-nodes - to spread existing shards thereby reducing impact of relocations, is not an option which we can choose
Please let me know if any additional details are required