Bringing this github conversation to this forum: https://github.com/elastic/elasticsearch/issues/14340
What is the best way to create a Master Node that can vote in a new Master Election, but itself will never get elected to be the cluster Master.
Option 1: Create a Zen Voting Only role
Option 2: Create a Zen weighted point system - where electable nodes get more points in the election
I realize this may not be possible today, but any ideas in carrying this suggestion into the roadmap would be great.
Problem:
The standard HA configuration of 3 Master nodes, will tolerate the failure of 1 master node.
2 Master nodes are required to achieve a quorum to elect a new master (and to avoid split-brain):
discovery.zen.minimum_master_nodes: 2
This requires 3 fault tolerant Zones...
If you only build in 2 fault tolerant Zones, you cannot distribute the master nodes sufficiently:
Zone A - Master1
Zone B - Master2, Master3
If you lose Zone A, you are cool. Master2 and Master3 will form a quorum and elect one of them as the master
If you lose Zone B, you are in trouble. Master1 will not have enough nodes to form a quorum. You can force the minimum_master_nodes to 1 and bootstrap Master1, but if Zone B's service is restored you will be in a split-brain status.
Business Justification:
In AWS, most regions offer a third Availability Zone, which certainly helps solve this problem. Because AWS charges for cross-AZ bandwidth, it means that the billable bandwidth can double, as all data nodes could be working with a cross-AZ master node. Today, only 50% of the nodes communicate across-AZ (as the rest talk to the master locally).
Others have also suggested further cost savings by utilizing a smaller class hardware for the Voting-only node, as it will never need to aggregate requests.
Just some ideas, would love to hear your thoughts.