Lost all index suddenly

Hi guys,
I do not know why and how, but on 31/12 I've lost all the index , 1TB of index.
I have 3 nodes and on all of them I have the same issue,
Any ideas? someone knows something?

Regards
Carmine

Do you have the elasticsearch cluster logs, in particular the master ones, covering that period?

Yes I do, I have a cluster ELK+Redis

Carmine, I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name).
This would be the first place to check to understand why these indices have disappeared.

Yes understood, no I dont have a Elasticsearch cluster

Yes I do, I have a cluster ELK+Redis
Yes understood, no I dont have a Elasticsearch cluster

you do or you don't?

Yo man, you're confusing me :joy:, I have 4 nodes configured like a cluster, I dont understand what do you mean here

I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name).

Thanks in advantage

Have a read at the basics perhaps it will clear confusion

The Basics

Main Elasticsearch logs are written to ES_HOME/logs/[cluster_name].log file. For this file the default level is INFO, thus being sufficient for a rather moderate amount of information and, at the same time, not create a huge log file.

that is the file(s) you want to check.

Yes, thanks a lot :grin:, I've seen it before, and this is my output

[2016-12-30 00:00:01,441][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2016-12-30 00:00:01,767][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:01,769][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic)
[2016-12-30 00:00:02,969][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:04,047][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:53,727][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:01:21,381][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [mysql-error] (dynamic)
[2016-12-30 03:20:18,247][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 03:30:02,601][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.27] deleting index
[2016-12-30 08:38:10,445][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 09:48:35,035][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [jetty] (dynamic)
[2016-12-30 10:48:20,159][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic)
[2016-12-31 00:00:00,618][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2016-12-31 00:00:00,814][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:00:00,815][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic)
[2016-12-31 00:00:02,952][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:00:12,350][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:01:21,075][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [mysql-error] (dynamic)
[2016-12-31 00:06:43,654][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 02:39:51,829][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 03:30:03,397][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index
[2016-12-31 04:23:15,799][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [jetty] (dynamic)
[2016-12-31 06:37:16,910][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 09:13:45,504][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 10:17:20,024][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic)
[2017-01-01 00:00:00,710][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2017-01-01 00:00:00,943][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:00:00,944][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic)
[2017-01-01 00:00:04,076][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:00:23,679][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:01:21,274][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [mysql-error] (dynamic)
[2017-01-01 00:06:43,980][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:37:09,127][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 03:30:02,888][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.29] deleting index
[2017-01-01 21:06:58,120][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic)

what this shows leads me to think you have curator or a cron-like job set up to delete indices everyday at 3:30 AM

I dont have anything, btw I've lost the index at 01:00am see the picture

I dont have anything

I take that means you don't have any cron job set up and I'd suggest to double check that.
The logs show that a DELETE is being called by someone, you could track this down using security auditing.
There is no auto-functionality to eliminate indices within Elasticsearch.

What timezone is this? CET? If so, was it the leap second?

yes, I have curator configured

curator --host #IP delete indices --older-than 180 --time-unit days --timestring '%Y.%m.%d'

that's it

1 Like

no UTC

Wed Jan 4 14:30:31 UTC 2017

What do you mean?

yes, I have curator configured

I see, however looking at this line for example

[2016-12-31 03:30:03,397][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index

I'd assume curator is configured to delete indices older than 3 days , not 180...

Do you enable "action.destructive_requires_name"?
It prevent removing indices through regex / "*"

No I dont, btw where is this parameter? /etc/elasticsearch/elasticsearch.yml?

maybe is enabled

Hi,

You have to set this param in the elasticsearch.yml file
https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html

Take a look at the doc here
https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-delete-index.html
It's about action.destructive_requires_name param

I've never configured, and there is not this param.
Thanks for the link