Find Cluster information Index

Elastic Stack Elasticsearch
1

Hi Team,

We have 3 nodes cluster for Elasticsearch and all of them are master eligible + data node. If two of ES nodes were down the third ES node is not able to find master node which is expected.

Once this happens (two ES goes down), we wanted to make the third ES node as standalone master node with minimum config changes in elasticsearch.yml file.

When we tried doing it (modified discovery.seed_hosts and cluster.initial_master_nodes), but somehow the ES node still keeps the 3 nodes cluster information in index and never gets started (ERROR: master not discovered or elected yet, an election requires at least 2 nodes with ids from... ).

How to delete only the index that keeps the cluster information?
How to find which index keeps the cluster information only so that delete the required index by not touching the data set?

Note: Deleting the entire data node/folder solves our problem while doing so, we need to re-index the node and that takes some time to sync with database.

Regards | Bibhu

2

In a cluster of 3 nodes, a quorum is 2.

If you have only 1 node, then you do not have a quorum and there is no safe way to proceed.
It is possible that the 1 node that is still running is out of date and does not have the latest cluster state. There is no way for that node to know whether this is the case or not.

Elasticsearch makes every effort to prevent you from accidentally running with a cluster that is unsafe. We never want anyone to lose data because they started the wrong node, or tweaked a cluster setting without realizing the impact.

If you truly cannot recover either of the other 2 nodes and want to make a best effort to recover the cluster with the single node that you have, then you should look at unsafe bootstrap in the elasticsearch-node tool.

There is no such index - the cluster state is not stored in an index.
And even if there was, if there is no elected master node then there is no way to delete an index.

1 Like