# Elasticsearch master pods failing master not discovered or elected yet, an election requires at least 2 nodes **URL:** \ **Category:** Community Ecosystem\ **Created:** [August 27, 2024, 5:54am UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591 "2024-08-27T05:54:03Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![gaurav671](https://avatars.discourse-cdn.com/v4/letter/g/c67d28/32.png) [@gaurav671](https://discuss.elastic.co/u/gaurav671)\ **Post date:** [August 27, 2024, 5:54am UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591/1 "2024-08-27T05:54:03Z")
Hello, I'm facing an issue with my elasticsearch cluster. I have 15 pods running in multiple nodes on my kubernetes cluster. 3 elasticsearch master pod, 12 elasticsearch-data pods are running . We are facing issue with Master pods failing as only 1 master pod is running and it is not identifying my remaining 2 pods and cluster is in async state. Please find below logs ```auto [2024-08-27T05:52:13,612][WARN][o.e.c.c.ClusterFormationFailureHelper] [twamp-es-master-1] master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{twamp-es-master-1}{iDt5Fo70ShKuNgEaopAtjQ}{TYFK2HM1TWW0i-0cQhCPIQ}{twamp-es-master-1}{192.168.14.47}{192.168.14.47:9300}{mr}{8.9.1}, {twamp-es-master-0}{k4_DXeaGQY6cjpOyMcqkgg}{ZBOZCjtMTTGKlr8SVXfeiQ}{twamp-es-master-0}{192.168.7.158}{192.168.7.158:9300}{mr}{8.9.1}, {twamp-es-master-2}{kBAN4EF7TzaUQmSsODP8aA}{jV7DVxERQLqba5TsMvsRfQ}{twamp-es-master-2}{192.168.18.23}{192.168.18.23:9300}{mr}{8.9.1}]; discovery will continue using [192.168.18.23:9300, 192.168.7.158:9300] from hosts providers and [{twamp-es-master-1}{iDt5Fo70ShKuNgEaopAtjQ}{TYFK2HM1TWW0i-0cQhCPIQ}{twamp-es-master-1}{192.168.14.47}{192.168.14.47:9300}{mr}{8.9.1}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.9/discovery-troubleshooting.html at org.elasticsearch.server@8.9.1/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:316) at org.elasticsearch.server@8.9.1/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:355) at org.elasticsearch.server@8.9.1/org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:293) at org.elasticsearch.server@8.9.1/org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:642) at org.elasticsearch.server@8.9.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1623) ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 27, 2024, 6:36am UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591/2 "2024-08-27T06:36:27Z")
Do all the master nodes have persistent storage? How are they configured? ---
**Author:** ![gaurav671](https://avatars.discourse-cdn.com/v4/letter/g/c67d28/32.png) [@gaurav671](https://discuss.elastic.co/u/gaurav671)\ **Post date:** [August 27, 2024, 6:40am UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591/3 "2024-08-27T06:40:25Z")
Yes all the master node has persistent storage . we actually wants to expand this nodes capacity hence we deleted the pvc and it restarted with updated expanded capacity. But again after that master pods giving us error, "master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial\_master\_nodes] is empty on this node:" ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 27, 2024, 8:48am UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591/4 "2024-08-27T08:48:56Z")
> [@gaurav671](#): > > `master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node` It looks to me like the cluster state that was saved on disk which contained the information about other nodes have been lost and you have apparently removed the initial master nodes config from the config (as recommended). I suspect you need to set the cluster up again as a new cluster, which will cause data los unless you have snapshot available. ---
**Author:** ![gaurav671](https://avatars.discourse-cdn.com/v4/letter/g/c67d28/32.png) [@gaurav671](https://discuss.elastic.co/u/gaurav671)\ **Post date:** [August 27, 2024, 11:05am UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591/5 "2024-08-27T11:05:01Z")
Any solution we have to recover data if we have shards available in pv pvc , we dont have snapshots available ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 27, 2024, 11:24am UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591/6 "2024-08-27T11:24:40Z")
I believe the [elasticsearch-node](https://www.elastic.co/guide/en/elasticsearch/reference/current/node-tool.html) utility was designed for this type of unsafe recovery, but I have no idea how you would be able to use this with a cluster on Kubernetes. As I believe you have lost all master nodes it is likely the data is lost and can not be recovered. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [August 27, 2024, 12:49pm UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591/7 "2024-08-27T12:49:59Z")
> [@gaurav671](#): > > Yes all the master node has persistent storage . we actually wants to expand this nodes capacity hence we deleted the pvc and it restarted with updated expanded capacity. But again after that master pods giving us error, > "master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial\_master\_nodes] is empty on this node:" I do not know much about kubernetes, but isn't the pvc the persistent storage? If you deleted it, you deleted the data for the node. The error is consistent with elasticsearch starting with an empty data dir. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 24, 2024, 12:50pm UTC](https://discuss.elastic.co/t/elasticsearch-master-pods-failing-master-not-discovered-or-elected-yet-an-election-requires-at-least-2-nodes/365591/8 "2024-09-24T12:50:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.