# Master not discovered or elected yet, an election requires a node with id \[F-Tn-Q6vQuKE0Fgi5qtUMg\] + 503 master not discovered exception
**URL:** https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949
**Category:** Elasticsearch
**Created:** [March 21, 2024, 10:20pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949 "2024-03-21T22:20:43Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 21, 2024, 10:20pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/1 "2024-03-21T22:20:43Z")
I have a elasticsearch cluster and it was work well before. But yesterday I accidentally deleted the Master node that has been elected. After this, the another master node cannot be elected by other nodes, at the same time, I got the error(below) on it's logging file:
```auto
master not discovered or elected yet, an election requires a node with id [F-Tn-Q6vQuKE0Fgi5qtUMg], have only discovered non-quorum [{OPT __Master2}{ezEK9jukQTGtVW2DP3cSjA}{tHNwll8bTlmlmNes1HJPJg}{OPT__ Master2}{192.168.1.30}{192.168.1.30:9801}{m}];
```
and i wanted fix this by using the /\_cluster/voting\_config\_exclusions API , but I got the another erro like below:
```auto
503 master not discovered exception
```
How can i do ?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 22, 2024, 6:38am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/2 "2024-03-22T06:38:43Z")
See these docs for help:
> **[Troubleshooting discovery | Elasticsearch Guide \[8.12\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-troubleshooting.html#discovery-no-master)**
In particular:
> If the logs or the health report indicate that Elasticsearch can’t discover enough nodes to form a quorum, you must address the reasons preventing Elasticsearch from discovering the missing nodes. The missing nodes are needed to reconstruct the cluster metadata. Without the cluster metadata, the data in your cluster is meaningless. The cluster metadata is stored on a subset of the master-eligible nodes in the cluster. If a quorum can’t be discovered, the missing nodes were the ones holding the cluster metadata.
>
> Ensure there are enough nodes running to form a quorum and that every node can communicate with every other node over the network. Elasticsearch will report additional details about network connectivity if the election problems persist for more than a few minutes. If you can’t start enough nodes to form a quorum, start a new cluster and restore data from a recent snapshot.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2024, 6:49am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/3 "2024-03-22T06:49:43Z")
How many master eligible nodes did your cluster initially have? Did you follow [the guidelines in the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/high-availability-cluster-small-clusters.html) when setting up the cluster? Which version of Elasticsearch are you using?
---
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 22, 2024, 6:52am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/4 "2024-03-22T06:52:30Z")
I can't getting back (recovery) the missing node and the cluster's meta data on it, I have tried many methods to recover it from the hard disk , but failed.
Now I follow what you said and checked the repository directory I saved as you said, and found that there were no snapshots. There were only some indicators and UUID directories in each repo like below:
```auto
(base) [root@hm-194 es-repo]# tree -d -L 3
.
├── backups
│ ├── market_subjects
│ │ └── indices
│ ├── patents
│ │ └── indices
│ └── SLRC
│ ├── indices
│ └── tests-yC7oYRGiRgS_uK20H-A7ug
└── long_term_backups
└── old_market_subjects_data
└── indices
11 directories
```
in this , the "market\_subjects", "patents" and "old\_market\_subjects\_data" is the repo name that I had created before. but the "SLRC" is the cluster name .
I didn't found any snapshot file , but I had seen the generated snapshot files list by my policy , so is it correct the data under the `repo.path` like above? or missing something ?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 22, 2024, 6:54am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/5 "2024-03-22T06:54:03Z")
I think [this issue](https://github.com/elastic/elasticsearch/issues/106639) answers Christian's questions: you only had two master-eligible nodes. But [as the docs say](https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-design.html#high-availability-cluster-design):
> A resilient cluster needs three master-eligible nodes so that if one of them fails then the remaining two still form a majority and can hold a successful election.
Unfortunately if you can't bring the node back online you'll need to build a new cluster and restore your data from backups.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2024, 6:57am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/6 "2024-03-22T06:57:00Z")
> [@Haoke98](#):
>
> I didn't found any snapshot file , but I had seen the generated snapshot files list by my policy , so is it correct the data under the `repo.path` like above? or missing something ?
If you have multiple nodes [Elasticsearch snapshots requires a shared filesystem repository](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/snapshots-filesystem-repository.html), e.g. NFS. Is this what you have configured and mounted on your nodes as the repo path?
---
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 22, 2024, 6:57am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/7 "2024-03-22T06:57:09Z")
I had six data node , two master node before, but yesterday I had deleted one data node and one master node accidentally. The version of the all node is 8.3.3
---
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 22, 2024, 7:07am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/8 "2024-03-22T07:07:50Z")
I was originally thinking about whether there was any way to force the election of my other master, but now the answer is obvious. The other master node does not have the meta information of the cluster, so this is not feasible.
Now I just want you to help me confirm. Can I use the snapshots repo path I showed you above to recover data? Because I saw that even if there is no specific snapshot file, there is a lot of data stored in the indicies folder. Can I use them to restore my indexes on the newly created cluster? If it is possible, how to do it? I looked through the Elastic official documentation and only snapshots can be used to restore it, but after my attempts, I created a new cluster , and this `repo.path` is set and is empty in the snapshots list of kibana.
---
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 22, 2024, 7:13am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/9 "2024-03-22T07:13:44Z")
Yes, I had configured it on the ./config/elasticsearch.yml on every node ( master / data ).
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2024, 7:17am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/10 "2024-03-22T07:17:14Z")
@DavidTurner This is unfortunately not an uncommon issue even though it is covered by the docs. As far as I know there are no scenarios where bootstrapping a new cluster with 2 master-eligible nodes is recommended. Given that Elasticsearch does add a number of bootstrap checks for what is deemed to be production clusters, would verifying that the number of initial master nodes is not 2 be a good candidate for a new bootstrap check? If we wanted to still allow this for some reason, might it be suitable to then have the user explicitly enable an "unsafe operation mode" through a configuration setting? Maybe it might be useful to also log a warning if there is only a single master eligible node in the cluster that is not a single-node cluster?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2024, 7:25am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/11 "2024-03-22T07:25:44Z")
What type of storage are you using for the repo? Is it an NFS mount?
---
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 22, 2024, 7:28am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/12 "2024-03-22T07:28:49Z")
lvm but, the lvm didn't have any snapshots, so I tried the ext4magic is failed.
```auto
(base) [root@hm-194 home]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 60T 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 60T 0 part
├─centos-root 253:0 0 50G 0 lvm /
├─centos-swap 253:1 0 4G 0 lvm [SWAP]
└─centos-home 253:2 0 60T 0 lvm /home
nvme0n1 259:0 0 931.5G 0 disk /ssd1
nvme1n1 259:1 0 931.5G 0 disk /ssd2
(base) [root@hm-194 home]#
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2024, 7:32am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/13 "2024-03-22T07:32:37Z")
That does not look like a shared filesystem. Is it?
For Elasticsearch snapshots to work they need shared storage, NFS storage accessible by all nodes over the network, so that files written by one node can be read from the repo by all other nodes. Having local directories under the same path on different machines does not work.
---
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 22, 2024, 7:40am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/14 "2024-03-22T07:40:04Z")
Oh cry, I really didn’t know this was the case before. Is there really no other way to recover these data?
---
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 22, 2024, 7:42am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/15 "2024-03-22T07:42:20Z")
Anyway, Thank you( @DavidTurner , @Christian_Dahlqvist ) very much for taking the time to help me figure it out.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 22, 2024, 7:58am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/16 "2024-03-22T07:58:05Z")
> [@Christian\_Dahlqvist](#):
>
> Given that Elasticsearch does add a number of bootstrap checks for what is deemed to be production clusters, would verifying that the number of initial master nodes is not 2 be a good candidate for a new bootstrap check?
Bootstrap checks run far too early in startup to know how many nodes there are in the cluster unfortunately, but you're right, it'd be good to have something in this area. I think these days we could add this check to the health report so I opened [#106640](https://github.com/elastic/elasticsearch/issues/106640) to suggest that.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2024, 8:20am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/17 "2024-03-22T08:20:41Z")
At bootstrap time you would know how initial master nodes is configured though, so you could check that this is not exactly 2 nodes. This would likely be enough catch a significant number of cases early on. You could still add it to the health report, but I am afraid this is likely to be overlooked just the docs often are.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 22, 2024, 8:41am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/18 "2024-03-22T08:41:47Z")
> [@Christian\_Dahlqvist](#):
>
> At bootstrap time you would know how initial master nodes is configured though, so you could check that this is not exactly 2 nodes.
Yeah but `cluster.initial_master_nodes` should only be set the first time the cluster starts, and often folks will get to a 2-node cluster by growing from a one-node cluster, so I don't think this will catch enough cases.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2024, 8:51am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/19 "2024-03-22T08:51:18Z")
There is the [elasticsearch-node tool](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/node-tool.html) that may be able to help you, but I have myself never had to use it.
---
### Author:  [@Haoke98](https://discuss.elastic.co/u/Haoke98)
#### Post date: [March 22, 2024, 4:51pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949/20 "2024-03-22T16:51:41Z")
Very appreciate your help, I had fix my problem by the way you suggest.
Although some shards cannot be recovered due to my stupid behavior ( I did not set the number of replicas, snapshot settings and repo configuration correctly ). But it has already saved my life.
In order to facilitate more people who have the same problem as me, I plan to record my steps here:
1. First, I deactivated all nodes in my catastrophically damaged cluster.
2. Secondly, I picked the data node that occupied the largest storage space from my damaged cluster, first modified its configuration, changed its role to `[master,data]` and used the `elasticsearch-node` tool. [Unsafe cluster bootstrapping](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/node-tool.html#node-tool-unsafe-bootstrap), so I got a new cluster, but the data and ID of my node have not changed.
In fact, some data can be recovered here, but it is still incomplete. This depends on your copy and chunking.
3. Immediately afterwards, start the remaining nodes. In fact, more accurately, you should start moving nodes, that is, migrating nodes from the damaged cluster to new nodes. This process requires the [detach-cluster](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/node-tool.html#node-tool-detach-cluster) operation.
First execute the `./bin/elasticsearch-node detach-cluster` command, then modify the discover in the configuration to point to the master node started in the first step, and then start the current node.
- **I hope everyone can gain something and help from it like I did** 😇
[Next page](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-an-election-requires-a-node-with-id-f-tn-q6vquke0fgi5qtumg-503-master-not-discovered-exception/355949.md?page=2)