When 2 of 3 masters die, how to restore the cluster?

Elastic Stack Elasticsearch
1

Hi, i am using ES version 7.17.0

My cluster consists of
3 master-eligible
2 data + ingest
2 Coordinating
1 kibana

and they are running on AWS EC2 as docker container. 1 EC2 has 1 container

And i am testing availability and restoring method of ES

When 2 master-eligibles are down and if those 2 masters can't be restored forever or for some time(ex : network issue), then there is only 1 master and other nodes. And cluster will be down and not work

In this case, can i restore cluster?

I tried adding 3 more master eligible nodes but they can't join to the cluster since existing cluster is not discovered yet due to not satisfying QUORUM

Is there any way to exclude 2 down masters from cluster? Or any way to restore cluster?

2

This is covered in these docs:

Without the cluster metadata, the data in your cluster is meaningless. The cluster metadata is stored on a subset of the master-eligible nodes in the cluster. If a quorum can’t be discovered, the missing nodes were the ones holding the cluster metadata. [...] If you can’t start enough nodes to form a quorum, start a new cluster and restore data from a recent snapshot.

2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.