How to removes master-eligible nodes from elasticsearch cluster when master is not discovered

I have an Elasticsearch cluster with three master nodes. Two of my master nodes VM's got corrupted so I am trying to remove them so that I can add newly created master node but I am unable to do so using the Voting configuration exclusions API.

Elasticsearch version: 7.15.2

POST http://localhost:9200/_cluster/voting_config_exclusions?node_names=master-vc2crtp1511022np,master-vc2crtp1511023np

Is returning

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

In my master node logs:

[2021-12-08T03:29:02,965][WARN ][o.e.c.c.ClusterFormationFailureHelper] [master-vc2crtp1511021np] master not discovered or elected yet, an election requires at least 2 nodes with ids from [hAL0sIjfRnmzcSKMWeeBmQ, FsXqXkNXSD6Yr2yRkGutEw, 57c_rEV8TMO7KfuUF4RfpQ], have only discovered non-quorum [{master-vc2crtp1511021np}{57c_rEV8TMO7KfuUF4RfpQ}{k4occpnbQQ-BiCRIUFLy_Q}{vc2crtp1511021np.fmr.com}{10.XXX.114.125:9300}{m}, {master-vc2crtp1511022np}{nY5A07RZSNKC6i7kTfinzA}{-oXrG-6eR4Oh2wGBNSz4YA}{vc2crtp1511022np.fmr.com}{10.XX.236.176:9300}{m}, {master-vc2crtp1511023np}{_Jc7pxgeQVubtUV6oAOtQg}{QoRBbjMrTNGu5MUWt9hsjw}{vc2crtp1511023np.fmr.com}{10.XXX.112.235:9300}{m}]; discovery will continue using [10.XX.236.176:9300, 10.XXX.112.235:9300] from hosts providers and [{master-vc2crtp1511021np}{57c_rEV8TMO7KfuUF4RfpQ}{k4occpnbQQ-BiCRIUFLy_Q}{vc2crtp1511021np.fmr.com}{10.XXX.114.125:9300}{m}] from last-known cluster state; node term 1239, last-accepted version 13755 in term 1239

If you've lost half or more of your master-eligible nodes then you have unfortunately lost the latest cluster state, without which the contents of your cluster are meaningless. If you can't recover at least one of those nodes then you will need to restore this cluster from a snapshot.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.