Indices got deleted suddenly

Elastic Stack Elasticsearch
1

We have an AWS Managed ES Cluster, and 4 indices in them. Our data was there in all the indices till 31st December, 2019. However, when we returned on 2nd Jan, 2020. We saw all the indices were deleted! This is really terrifying for us. I quickly did a _cat/indices with creation time, and I noticed that the index kibana_1 had creation time at 31st December, 8:40 AM GMT and one index had creation time at 31st December, 1 PM GMT. Some other indices were created after that time due to ingestion triggers from our product.

But that shouldn't be the case, since our cluster was hosted 1-1.5 months before. We never stopped the AWS ES Cluster from the point it was up. We tried to know the uptime of the cluster assuming any restart would have impacted the indices in any way but didn't find a way to see know the uptime of the cluster.

We did revert to a snapshot that we had on 31st December. But we did lose all the 2 days data. I really would like to know under what conditions something like this will happen. We are going to production with the whole product in a few weeks and assuming this can happen again is scaring me now.

Any thoughts and ideas of how this happened and how to prevent it from happening in future?

2

Do you have any form of access control on your cluster?

3

You should also check the logfile on the master node to find out exactly when each index was deleted, look for something like:

[2020-01-04T01:00:00,057][INFO ][o.e.c.m.MetaDataDeleteIndexService] [master-1] [.monitoring-kibana-6-2019.12.27/ZqSOjqh1WFSVqswXSiy2dg] deleting index

It doesn't tell you why or who made the deletion but it gives you the exact time, which could be useful for further research.

4

What is the size of the cluster? As you are using AWS ES I would recommend checking with AWS support.

5

Problem is, we haven't enabled logging on the AWS Cluster. And we're clueless because of that now.

6

Probably less than 5 GB including all indices, since not much data went in during the development and testing stage.

7

As of now we haven't enabled any such control. But only a couple of folks have access to cluster and none of them ran a delete request on indices.

8

How many nodes do you have in the cluster?

9

We have just one node. And the data nodes storage type is EBS as well.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.