# Master not discovered, removed nodes have been totally destroyed **URL:** \ **Category:** Elasticsearch\ **Created:** [November 1, 2019, 10:21am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143 "2019-11-01T10:21:43Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![hackerwin7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackerwin7/32/43223_2.png) [@hackerwin7](https://discuss.elastic.co/u/hackerwin7)\ **Post date:** [November 1, 2019, 10:21am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/1 "2019-11-01T10:21:43Z")
Hi, I have removed half of nodes of cluster, and the removed nodes and machines have been totally destroyed. the existing cluster log like: ```auto master not discovered or elected yet, an election requires at least nodes with ids from ... have discovered [] which is not quorum; discovery will continue using... ``` After seeking the docs, there is a way to reinstate removed nodes to let the node discovery the master eligible to form a quorum. However, the removed nodes have been permanently removed, can not reinstate, we hope to don't delete the whole path.data to re-build the cluster, we want to keep the indices shard data in path.data, and only wipe the previously saved master eligible states in path.data, voting configuration in the path.data. Is there any ways to achieve this? ---
**Author:** ![hackerwin7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackerwin7/32/43223_2.png) [@hackerwin7](https://discuss.elastic.co/u/hackerwin7)\ **Post date:** [November 1, 2019, 10:25am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/2 "2019-11-01T10:25:25Z")
@DavidTurner Would you have interest for this? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 1, 2019, 10:30am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/3 "2019-11-01T10:30:18Z")
Have a look at [this thread](https://discuss.elastic.co/t/handling-loss-of-master-quorum-in-es7/191443) which should apply as you no longer have quorum of master nodes. ---
**Author:** ![hackerwin7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackerwin7/32/43223_2.png) [@hackerwin7](https://discuss.elastic.co/u/hackerwin7)\ **Post date:** [November 1, 2019, 11:28am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/4 "2019-11-01T11:28:57Z")
Hi, After reading the thread, I add many new empty nodes to let the quorum matched, the master is elected by the added new nodes, but the existing old nodes failed to send join to new master. log below: ```auto join validation on cluster state with a different cluster uuid ZTfGFra-QrS4FpdLjlQGFQ than local cluster uuid rpjd3TKOTyeqpbCgBVFynQ, rejecting ``` How to wipe the cluster states info saved in `path.data` and keep the shard index data and let the old nodes to join and form a cluster ? ---
**Author:** ![hackerwin7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackerwin7/32/43223_2.png) [@hackerwin7](https://discuss.elastic.co/u/hackerwin7)\ **Post date:** [November 1, 2019, 11:36am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/5 "2019-11-01T11:36:26Z")
our elasticsearch version : 7.3.0 we have no snapshot, and the removed nodes can not come back to cluster. How to let the old nodes with the path.data and form a quorum and cluster again ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 1, 2019, 11:58am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/6 "2019-11-01T11:58:43Z")
I am not sure that is possible. See [the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/modules-discovery-quorums.html) for further details. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 1, 2019, 12:13pm UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/7 "2019-11-01T12:13:32Z")
If you lost half or more of the master nodes then you have lost the cluster metadata, without which Elasticsearch cannot correctly interpret the data held in all the shards. Since you don't have a snapshot either the only sensible thing to do is rebuild the cluster from data held elsewhere as best as you can. ---
**Author:** ![hackerwin7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackerwin7/32/43223_2.png) [@hackerwin7](https://discuss.elastic.co/u/hackerwin7)\ **Post date:** [November 6, 2019, 11:49am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/8 "2019-11-06T11:49:58Z")
Hi, After reading this [blog](https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch) > For example, an Elasticsearch 7.0 cluster will not automatically recover if half or more of the master-eligible nodes are permanently lost. It is common to have three master-eligible nodes in a cluster, allowing Elasticsearch to tolerate the loss of one of them without downtime. If two of them are permanently lost then the remaining node cannot safely make any further progress. > Versions of Elasticsearch before 7.0 would quietly allow a cluster to recover from this situation. Users could bring their cluster back online by starting up new, empty, master-eligible nodes to replace any number of lost ones. An automated recovery from the permanent loss of half or more of the master-eligible nodes is unsafe, because none of the remaining nodes are certain to have a copy of the latest cluster state 7.x seems more restricted to re-form the cluster. according to this: > If neither of these recovery actions is possible then the last resort is the [`elasticsearch-node` unsafe recovery tool](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/node-tool.html). This is a command-line tool that a system administrator can run to perform unsafe actions such as electing a stale master from a minority. By making the steps that can break consistency very explicit, Elasticsearch 7.0 eliminates the risk of unintentionally causing data loss through a series of unsafe operations. **unsafe bootstrap** maybe make sense ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 4, 2019, 11:49am UTC](https://discuss.elastic.co/t/master-not-discovered-removed-nodes-have-been-totally-destroyed/206143/9 "2019-12-04T11:49:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.