# How to recover cluster when 2 master nodes have been lost **URL:** \ **Category:** Elasticsearch\ **Created:** [April 7, 2021, 1:16pm UTC](https://discuss.elastic.co/t/how-to-recover-cluster-when-2-master-nodes-have-been-lost/269472 "2021-04-07T13:16:57Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![russinholi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russinholi/32/25665_2.png) [@russinholi](https://discuss.elastic.co/u/russinholi)\ **Post date:** [April 7, 2021, 1:16pm UTC](https://discuss.elastic.co/t/how-to-recover-cluster-when-2-master-nodes-have-been-lost/269472/1 "2021-04-07T13:16:57Z")
Hi everyone, I have the following situation, my cluster is formed by 3 master nodes and 3 data nodes, unfortunately due to a hardware problem we lost 2 master nodes at same time and the left one was not able to assume as master. And now when I recreated those 2 master nodes, with the same node names and IPs, they are not able to join into the cluster, and I start to get the following message when I start all 3 master nodes: `[masternode-2] received cluster state from {masternode-3}{MZkz2i3qQYCpivdepDtACA}{4naaRVrvQG6_aLXw5i9qsA}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}{cdhilmrstw}{ml.machine_memory=20994146304, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} with a different cluster uuid F_KaGI1GR2Kw2oPmS-czmA than local cluster uuid L4iw4uQhQiCStSUtadLjQg, rejecting` masternode-2 is the survival one, masternode-1 and masternode-3 are the ones we lost, it seems that the latest cluster state had masternode-3 as the master one and when it tries to join we get that message. Any hint on how to recover this cluster without losing the data? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 7, 2021, 2:54pm UTC](https://discuss.elastic.co/t/how-to-recover-cluster-when-2-master-nodes-have-been-lost/269472/2 "2021-04-07T14:54:47Z")
The cluster metadata is stored on a majority of the master nodes, so 2 of the 3, and it was lost when those nodes died. Without this metadata, the data in your cluster is unfortunately not meaningful. You'll need to restore it from a snapshot. ---
**Author:** ![russinholi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russinholi/32/25665_2.png) [@russinholi](https://discuss.elastic.co/u/russinholi)\ **Post date:** [April 7, 2021, 3:02pm UTC](https://discuss.elastic.co/t/how-to-recover-cluster-when-2-master-nodes-have-been-lost/269472/3 "2021-04-07T15:02:53Z")
Thanks David, unfortunately we don't have a snapshot (yeah I know we should 😅 ), do you think I could use the strategy of detaching them from the cluster using elasticsearch-node tool, as you suggested on [Recover data after the lost of master [7.1.1] - #5 by DavidTurner](https://discuss.elastic.co/t/recover-data-after-the-lost-of-master-7-1-1/187775/5) ? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 7, 2021, 3:05pm UTC](https://discuss.elastic.co/t/how-to-recover-cluster-when-2-master-nodes-have-been-lost/269472/4 "2021-04-07T15:05:07Z")
You can _try_ but as I said there, it comes with no guarantees, it might not even warn you about any lost data. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 5, 2021, 3:05pm UTC](https://discuss.elastic.co/t/how-to-recover-cluster-when-2-master-nodes-have-been-lost/269472/5 "2021-05-05T15:05:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.