# Master nodes lost **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [July 29, 2020, 2:33pm UTC](https://discuss.elastic.co/t/master-nodes-lost/243075 "2020-07-29T14:33:51Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Marcus\_Simonsen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcus_simonsen/32/54552_2.png) [@Marcus\_Simonsen](https://discuss.elastic.co/u/Marcus_Simonsen)\ **Post date:** [July 29, 2020, 2:33pm UTC](https://discuss.elastic.co/t/master-nodes-lost/243075/1 "2020-07-29T14:33:51Z")
Hello Users + Experts. Due to unfortunate operator error the PVs of the master nodes were deleted, due to a out of disk error from the master nodes. We had 3 dedicated master nodes and 12 data nodes. The data nodes are now can not locate the master after provisioning new master nodes (and yes that node set was scaled down to zero sadly). ```auto {"type": "server", "timestamp": "2020-07-29T14:25:20,697Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "eck-prod", "node.name": "eck-prod-es-data-nodes-2-3000-0", "message": "path: /_ilm/policy/filebeat-7.4.1, params: {name=filebeat-7.4.1}", "cluster.uuid": "vcExmqmWQqKJjBuIQ8b1TA", "node.id": "9CIgWcavQXedBXGEHPfpqA" , "stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null", "at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:218) [elasticsearch-7.5.2.jar:7.5.2]", "at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.5.2.jar:7.5.2]", "a ``` We are therefore left with 12 data nodes (from 4 node sets) and no ability to form a cluster. We have tried a few things to recover the cluster Following this guide: [https://www.elastic.co/guide/en/elasticsearch/reference/current/node-tool.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/node-tool.html) Which is clearly not applicable to kubernetes and eck. We have additionally thought about making 3 of the data nodes master eligible, and if possible triggering a master election... although we aren't sure if this is the right path. Any help or guidance appreciated!! ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [July 30, 2020, 11:31am UTC](https://discuss.elastic.co/t/master-nodes-lost/243075/2 "2020-07-30T11:31:20Z")
By far the **best option** would be to **restore from a snapshot into a new cluster**. If that is not possible you could try the following: - spin up new master nodes (I think you have done that already) - set `cluster.initial_master_nodes` explicitly to the new names of the new master nodes (this will generate a warning in the logs as this setting is usually managed by ECK) - now for the **dangerous** part. Please note that the following **can lead to data loss** and should only be your measure of last resort: - convince yourself that you have correctly configured `cluster.initial_master_nodes` and the new masters have formed a new cluster - now edit the spec of your data nodes adding an init container that will detach the data nodes from the old cluster and allow them to join the newly formed cluster. Again this is where you can lose data. I used the following container in an experiment: ```auto spec: initContainers: - name: detach command: - sh - -c - | printf "y\n" | bin/elasticsearch-node detach-cluster ``` - Make sure that the new pod specification with the addtional init container is used in the corresponding stateful set. If not wait for the operator to update the stateful set before you continue. - Now start with one of the data nodes and delete the corresponding pod. The pod will then be recreated by the stateful set controller using the new spec. - Check the pods logs and verify that it successfully joins the new cluster before proceding to repeat the same procedure with the other data pods. - Once all nodes have joined the new cluster remove the init container (important otherwise the pods will detach every time they restart) and also the `cluster.initial_master_nodes` setting again. I suggest to go through the procedure on a test cluster first to verify it works for your setup, before you try to apply to the cluster that has lost its master nodes. [EDIT] I have simplified the instructions a bit. The `cluster.initial_master_nodes` setting is not necessary on the data nodes. ---
**Author:** ![Marcus\_Simonsen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcus_simonsen/32/54552_2.png) [@Marcus\_Simonsen](https://discuss.elastic.co/u/Marcus_Simonsen)\ **Post date:** [July 30, 2020, 12:40pm UTC](https://discuss.elastic.co/t/master-nodes-lost/243075/3 "2020-07-30T12:40:26Z")
Many thanks Peter, We don't snapshot globally or any system indices, only specific indices, so that option is out. I had thought about the initial master node setting as well, but the detach cluster on initContainers was the missing ingredient - genius! -marcus ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 8:08am UTC](https://discuss.elastic.co/t/master-nodes-lost/243075/4 "2022-11-04T08:08:10Z")