Sudden deletion of data

Elastic Stack Elasticsearch
1

Elasticsearch version : 5.6.8

Plugins installed: [] No plugins

JVM version (java -version): 8

OS version (uname -a if on a Unix-like system): Linux Centos 7

Description of the problem including expected versus actual behavior:

I have an elastic search cluster of 1 node in my environment.

The cluster was running fine from months.

Today all of a sudden I saw no data in the data directory.

Just like that.

We can't even know why did it happen

We did not take any backup or snapshots.

Is there anyway to recover the lost data?

Elasticsearch logs:

[2018-03-18T20:55:32,950][WARN ][r.suppressed ] path: /_search/scroll, params: {pretty=} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.SearchScrollAsyncAction.onShardFailure(SearchScrollAsyncAction.java:269) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.search.SearchScrollAsyncAction.run(SearchScrollAsyncAction.java:158) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.search.SearchScrollAsyncAction.lambda$run$0(SearchScrollAsyncAction.java:110) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.6.8.jar:5.6.8] 
[2018-03-18T21:27:40,709][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2018.02.26/nKlg0SZ-QSqj9hOdQQI9Gw] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2018.03.09/gH4W1EwdQayWOJIa7vUXnw] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2017.12.14/Dfz4TS4rTnG-q_FsBcEpIg] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2017.12.07/bhhGQsF5TUmAYRa8gOTo_w] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2018.01.16/VoxuRIUcSVm8H9Ycchsteg] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2018.01.22/JwmOheLdQCCltcEpdubkcg] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2017.09.14/Hln9N7ViRW-_OONI0sEt8Q] deleting index

Kibana logs:

Mar 18 21:27:56 localhost.localdomain kibana[4946]:{"type":"log","@timestamp":"2018-03-18T21:27:56Z","tags" ["status","plugin:elasticsearch@5.6.8","error"],"pid":4946,"state":"red","message":"Status changed from green to red - all shards failed: [search_phase_execution_exception] all shards failed","prevState":"green","prevMsg":"Kibana index ready"}
Mar 18 21:27:56 localhost.localdomain kibana[4946]: {"type":"log","@timestamp":"2018-03-18T21:27:56Z","tags":["status","ui settings","error"],"pid":4946,"state":"red","message":"Status changed from green to red Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"} Mar 18 21:28:11 localhost.localdomain kibana[4946]: {"type":"log","@timestamp":"2018-03-18T21:28:11Z","tags":["status","plugin:elasticsearch@5.6.8","info"],"pid":4946,"state":"yellow","message":"Status changed from red to yellow - No existing Kibana index found","prevState":"red","prevMsg":"all shards failed: [search_phase_execution_exception] all shards failed"} Mar 18 21:28:11 localhost.localdomain kibana[4946]: {"type":"log","@timestamp":"2018-03-18T21:28:11Z","tags":["status","ui settings","info"],"pid":4946,"state":"yellow","message":"Status changed from red to yellow - Elasticsearch plugin is yellow","prevState":"red","prevMsg":"Elasticsearch plugin is red"}
Mar 18 21:28:12 localhost.localdomain kibana[4946]: {"type":"log","@timestamp":"2018-03-18T21:28:12Z","tags":["status","plugin:elasticsearch@5.6.8","info"],"pid":4946,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
Mar 18 21:28:12 localhost.localdomain kibana[4946]: {"type":"log","@timestamp":"2018-03-18T21:28:12Z","tags":["status","ui settings","info"],"pid":4946,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Elasticsearch plugin is yellow"}
2

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

3

Thank you, and I updated the post.

4

Thanks. I had to edit it a bit more but anyway it's more readable now.

We can't even know why did it happen

You can see that something/someone called the DELETE index API here:

[2018-03-18T21:27:40,709][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2018.02.26/nKlg0SZ-QSqj9hOdQQI9Gw] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2018.03.09/gH4W1EwdQayWOJIa7vUXnw] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2017.12.14/Dfz4TS4rTnG-q_FsBcEpIg] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2017.12.07/bhhGQsF5TUmAYRa8gOTo_w] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2018.01.16/VoxuRIUcSVm8H9Ycchsteg] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2018.01.22/JwmOheLdQCCltcEpdubkcg] deleting index 
[2018-03-18T21:27:40,710][INFO ][o.e.c.m.MetaDataDeleteIndexService] [nfc-server] [logstash-2017.09.14/Hln9N7ViRW-_OONI0sEt8Q] deleting index

Some options:

  • A batch is running on your system and ran at 2018-03-18T21:27:40 and started to remove some indices. It might be intentional, or a buggy script which ran DELETE * instead of DELETE indexName or a bad configuration of curator if you are using it.
  • Someone ran a DELETE * command by mistake on your system. Someone did that intentionally.

Is there anyway to recover the lost data?

No. Because:

We did not take any backup or snapshots.

It sounds like your system is not secured and may be someone has access to it.
Note that X-Pack security (official security commercial plugin by elastic) would prevent that and would also allow you to trace who is trying to access a forbidden API like DELETE Index API with audit logging feature.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.