Cluster recovery after deleting indices directory

Elastic Stack Elasticsearch
1

Hello team,
First of all, thank you for the product.

My question is: is there any way to recover a cluster after having removed the path.data/nodes/<node>/indices ? After having done that, my cluster is not able to find any master node. I have master_not_discovered_exception error. The credentials for the API are not working either.
If there's absolutely no way to recover that cluster and the data, how can I bootstrap a new cluster? I've changed the cluster.name option in every node and I'm still unable to bootstrap a new cluster.

Please help! Thanks in advance.
Regards,
Sergio.

2

Hi @Sergio_Peral_Aguiler,

You can only recover your indices/data if you have snapshot/backup of your data. There is no other way to recover your cluster in elasticsearch. However, for master_not_discovered_exception i need the logs to identify the problem.

could you please share the logs for the same?

can i know what error message are showing while starting elasticsearch cluster?

Thanks,
Manish

3

Hi @manish19,
Thanks for your response.
After deleting the indices folder, I think the credentials of the cluster were lost/corrupted and the API is throwing unauthorized error for every query. Then, I tried to set new credentials and got the following error:
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

Failed to determine the health of the cluster running at [http://ES_IP:9200](http://ES_IP:9200/)
Unexpected response code [503] from calling GET http://ES_IP:9200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

In the cluster logs I have:
[2019-09-26T07:59:49,594][WARN ][o.e.h.AbstractHttpServerTransport] [] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/127.0.0.1:40320}
Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
WARN ][o.e.c.c.ClusterFormationFailureHelper] [xxxxxxxxxxxxxx] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes

I can telnet between nodes through port 9300 and I can see the connections in netstat.

I understand the snapshot part, but I can't recover the snapshot because I can't use the API. How could I make the cluster work again (even though the data is lost)? Or how could I start a new cluster without reinstalling?

Thanks again.
Regards,
Sergio.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.