Elasticsearch master node replacment

[ali_tavakoli]

Hi guys

I run elasticsearch on 8 nodes with 2 different rack-id for shard replication awareness.
i have 4 nodes on rack-1 and other 4 nodes on rack-2 and i have 1 master eligible node on each rack.
so i have 2 master nodes and 6 data nodes. it's ok when i start all 8 node and master node elect and every thing is fine! when i stop master node i expect that second master eligible elect as master and cluster works fine but i have this error:
" master not discovered or elected yet, an election requires a ..."

Here is my config in master A:

cluster.name: S-cluster
node.name: node-a-1
node.attr.rack_id: rack_one
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: site
discovery.seed_hosts: ["192.168.100.20", "192.168.100.21", "192.168.100.22", "192.168.100.23", "192.168.100.25", "192.168.100.26", "192.168.100.27", "192.168.100.28"]
cluster.initial_master_nodes: ["192.168.100.20", "192.168.100.25"]

node.master: true
node.data: true

cluster.routing.allocation.awareness.attributes: rack_id

and here is my data node config:

cluster.name: S-cluster
node.name: node-a-2

node.attr.rack_id: rack_one

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.100.21

discovery.seed_hosts: ["192.168.100.20", "192.168.100.21", "192.168.100.22", "192.168.100.23", "192.168.100.25", "192.168.100.26", "192.168.100.27", "192.168.100.28"]
cluster.initial_master_nodes: ["192.168.100.20", "192.168.100.25"]

node.master: false
node.data: true

and same config for master B and other data node! master IP is 192.168.100.20 and 192.168.100.25.
Thanks for your help.

[Christian_Dahlqvist]

That is expected and the correct behaviour. Elasticsearch master election is based on consensus, so a majority of master eligible nodes are required in order to elect a master. The majority of 2 is 2, which is why a master can not be elected when 1 master eligible node goes missing. In order to be able to handle one master eligible node going down you therefore need at least 3 master eligible nodes in the cluster.

[ali_tavakoli]

Thanks for your quick reply. so what do u suggest? i run this cluster for fail over and fault tolerance. I want cluster works if one of my rack goes down! is this OK if i config 2 master eligible nodes on each rack?

[Christian_Dahlqvist]

That won’t help as the majority of 4 is 3. You generally need another node outside these racks.

[ali_tavakoli]

Can i change config manually when one of masters fail? is this OK if i change master eligible to healthy master node manually in all nodes and restart cluster? so i have a delay time to start cluster with all data.

[Christian_Dahlqvist]

If you have 4 master eligible nodes you can handle a single node going down, but not a full rack.

[DavidTurner]

@Christian_Dahlqvist is absolutely correct, but to explain the issue a bit more: rack A cannot tell the difference between being disconnected from rack B and rack B actually failing; similarly rack B cannot tell the difference between being disconnected from rack A and rack A actually failing. Thus if the two racks were disconnected from each other and each of them believed that this meant the other rack had failed then both racks would elect a master and continue independently. This is called a split-brain and results in data loss. Elasticsearch will not let this happen.

It is impossible to solve this using only two racks. That's not a constraint that Elasticsearch imposes, it's a fundamental property of distributed systems. You need at least one more master-eligible node, independent of these two racks, to act as a tie-breaker in the case where the two racks are disconnected from each other.

You should not attempt to override this manually, because this too can result in data loss. Adding an independent node is the only safe way to achieve what you ask.

[ali_tavakoli]

Thanks a you so much guys, thats why i love elastic; becasue of you and your support. So i add another only master eligible node to cluster outside the racks, and i think it dosent need resourse as other nodes because it's just master node and not data node.

[DavidTurner]

Yes that's right, it won't need as much disk space if it's not a data node.

When 7.3 is released you can also make it a voting-only master-eligible node to prevent it from becoming master, which reduces its resource requirements further too.

[ali_tavakoli]

Very nice thank you so much guys.