Recover data after the lost of master [7.1.1]

Elastic Stack Elasticsearch
1

Hi,

I have a cluster with 3 masters node and 3 data node.
I've lost my 3 masters at the same time, is there a way to recover the data ?
I've tried to promote my 3 data node to the master but I have this log:

master not discovered or elected yet, an election requires at least 2 nodes with ids from [LaOUswy8TgG4GCWhtGO0Eg, kYSxOZ2dQjOFjvxAkj2s9g, iSw9oU5XS5eCtPjT6NJaSQ], have discovered [{es-data2-1}{CtvxakqiQ42yQjgyiFA0Kg}{MynpI1rsSmCeUJs36aZlAw}{100.96.22.25}{100.96.22.25:9300}{ml.machine_memory=16636354560, ml.max_open_jobs=20, xpack.installed=true}, {es-data2-2}{SHqLeNMITnizAIG5sFFGHQ}{mGjI5RJRQV-j6NuQ353YZg}{100.96.11.243}{100.96.11.243:9300}{ml.machine_memory=16636346368, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [100.96.11.243:9300, 100.96.22.25:9300] from hosts providers and [{es-data2-0}{zxg1_3GvQtmgVwB4PmLRvg}{smM8qzBdSz6XVDZYZVfCJw}{100.96.5.242}{100.96.5.242:9300}{ml.machine_memory=16636354560, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5, last-accepted version 1142 in term 5" }

2

Configure 2 Nodes with Master.Allow them to communicate.So that it wont result in split brain issue.But you will have duplicate records in indexes.

3

Unfortunately the master nodes hold important information about the cluster that can't be rebuilt from elsewhere. The best thing to do is recover the master nodes - as long as at least two of the masters' data folders are intact you should be able to start nodes on these data folders and reinstate the cluster. If you can't recover the master nodes, your best bet is to build a new cluster and restore the data from a recent snapshot.

4

Thanks for your response.

My master's data folders are definitevely lost. And I have no snapshot. (I haven't realised that the data folder of master was so important and thanks docker & k8s for the reset of pod) So there is no way (even a hard way) to rebuild my index juste with the data folder of my data node ?

5

After losing the masters there's no guarantees that your data will come back intact, but you can try the elasticsearch-node tool, particularly elasticsearch-node detach-cluster. Please read the instructions carefully and understand the risks before proceeding.

For future reference, it's vitally important to use persistent volumes for the master-eligible nodes in the cluster, as well as for the data nodes.

6

Thanks very much !
Now, I can see my shards and indices. I'm checking data but it seem ok.
I've created a new cluster with a persistent volume for masters nodes. And I've installed the s3 plugin for snapshot.

Thanks again for your help.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.