Accidentally deleted all indices from the cluster. How do I restore the data?

Elastic Stack Elasticsearch
1

Hi,

I know we messed up big time by accidentally deleting all the indices from the cluster but I needed input regarding what is the proper procedure to restore it back.
While running the curator tool, accidentally gave the wrong action file thus resulting in all the indices being deleted from the cluster. This cluster had all x-pack settings configured, ssl settings configured, ldap and so on.
The issue now is that, though I do have a snapshot of the recent data present in one of the folders of the server (This path is not registered as a repository yet, though it contains the snapshot), the issue is in starting elasticsearch service again because the .security index has also been deleted thus we get an x-pack exception. I have stopped the ES, kibana and logstash services for now. I just want to know the right sequence of restoring the data from the snapshot I have, I have not restored a snapshot manually before so It would be really helpful if someone could just guide me regarding this.

2

The best option would be to setup a file realm and then use that to auth and restore from there.

3

@warkolm
Will I be able to restore the elastic credentials without having to re-do the entire x-pack password set up? Because I really need to restore the .security index with the recent elastic credentials rather than setting new password for it.

4

Once you restore you should be ok to use those details. But you will need to run through the setup for file realm to get you access back to Elasticsearch.

5

@warkolm I configured the file realm, used the users command to add a new user with superuser role and also commented the ldap realm in my yaml just in case. But even if I try to start elasticsearch services after the file realm changes in place, it still gives the error:
[2018-09-17T12:30:10,755][INFO ][o.e.x.s.a.AuthenticationService] [] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

I followed these steps to configure the realm:
https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-file-realm.html

6

I figured out the issue. I had to disable all the xpack settings and ssl settings as monitoring, watcher, etc seem to be using the elastic credential.

7

This is not correct. I don't know what issue you ran into, but internal services do not use the elastic user.

[2018-09-17T12:30:10,755][INFO ][o.e.x.s.a.AuthenticationService] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

This is not an error. It is an informational message that the incorrect password was provided for te elastic user, but it does not prevent the service from starting.

8

@TimV
I realize that but I had already stopped the services (kibana & logstash) that are connecting to elastic for the elastic user credentials, yet this incorrect password message still persisted. It was difficult to start the services because this message kept appearing every second in the logs so I was not able check what other logs were getting generated after my file realm changes. I needed re-configure the x-pack set up passwords and check if it was getting properly authenticated.
But I wasn't able to figure out where this info message kept coming from.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.