Lost all index suddenly

Carmine_Fabrizio · January 3, 2017, 11:20am

Hi guys,
I do not know why and how, but on 31/12 I've lost all the index , 1TB of index.
I have 3 nodes and on all of them I have the same issue,
Any ideas? someone knows something?

Regards
Carmine

nellicus · January 3, 2017, 11:22am

Do you have the elasticsearch cluster logs, in particular the master ones, covering that period?

Carmine_Fabrizio · January 3, 2017, 11:25am

Yes I do, I have a cluster ELK+Redis

nellicus · January 3, 2017, 11:27am

Carmine, I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name).
This would be the first place to check to understand why these indices have disappeared.

Carmine_Fabrizio · January 3, 2017, 11:34am

Yes understood, no I dont have a Elasticsearch cluster

nellicus · January 3, 2017, 11:35am

Yes I do, I have a cluster ELK+Redis
Yes understood, no I dont have a Elasticsearch cluster

you do or you don't?

Carmine_Fabrizio · January 3, 2017, 11:48am

Yo man, you're confusing me , I have 4 nodes configured like a cluster, I dont understand what do you mean here

I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name).

Thanks in advantage

nellicus · January 3, 2017, 11:52am

Have a read at the basics perhaps it will clear confusion

The Basics

Main Elasticsearch logs are written to ES_HOME/logs/[cluster_name].log file. For this file the default level is INFO, thus being sufficient for a rather moderate amount of information and, at the same time, not create a huge log file.

that is the file(s) you want to check.

Carmine_Fabrizio · January 3, 2017, 12:07pm

Yes, thanks a lot , I've seen it before, and this is my output

[2016-12-30 00:00:01,441][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2016-12-30 00:00:01,767][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:01,769][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic)
[2016-12-30 00:00:02,969][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:04,047][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:53,727][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:01:21,381][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [mysql-error] (dynamic)
[2016-12-30 03:20:18,247][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 03:30:02,601][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.27] deleting index
[2016-12-30 08:38:10,445][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 09:48:35,035][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [jetty] (dynamic)
[2016-12-30 10:48:20,159][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic)
[2016-12-31 00:00:00,618][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2016-12-31 00:00:00,814][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:00:00,815][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic)
[2016-12-31 00:00:02,952][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:00:12,350][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:01:21,075][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [mysql-error] (dynamic)
[2016-12-31 00:06:43,654][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 02:39:51,829][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 03:30:03,397][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index
[2016-12-31 04:23:15,799][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [jetty] (dynamic)
[2016-12-31 06:37:16,910][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 09:13:45,504][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 10:17:20,024][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic)
[2017-01-01 00:00:00,710][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2017-01-01 00:00:00,943][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:00:00,944][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic)
[2017-01-01 00:00:04,076][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:00:23,679][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:01:21,274][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [mysql-error] (dynamic)
[2017-01-01 00:06:43,980][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:37:09,127][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 03:30:02,888][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.29] deleting index
[2017-01-01 21:06:58,120][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic)

nellicus · January 3, 2017, 12:40pm

what this shows leads me to think you have curator or a cron-like job set up to delete indices everyday at 3:30 AM

Carmine_Fabrizio · January 3, 2017, 1:44pm

I dont have anything, btw I've lost the index at 01:00am see the picture

nellicus · January 3, 2017, 1:58pm

I dont have anything

I take that means you don't have any cron job set up and I'd suggest to double check that.
The logs show that a DELETE is being called by someone, you could track this down using security auditing.
There is no auto-functionality to eliminate indices within Elasticsearch.

jprante · January 4, 2017, 1:24am

What timezone is this? CET? If so, was it the leap second?

Carmine_Fabrizio · January 4, 2017, 2:30pm

yes, I have curator configured

curator --host #IP delete indices --older-than 180 --time-unit days --timestring '%Y.%m.%d'

that's it

Carmine_Fabrizio · January 4, 2017, 2:32pm

no UTC

Wed Jan 4 14:30:31 UTC 2017

What do you mean?

nellicus · January 4, 2017, 2:36pm

yes, I have curator configured

I see, however looking at this line for example

[2016-12-31 03:30:03,397][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index

I'd assume curator is configured to delete indices older than 3 days , not 180...

junior_h · January 4, 2017, 2:38pm

Do you enable "action.destructive_requires_name"?
It prevent removing indices through regex / "*"

Carmine_Fabrizio · January 4, 2017, 3:08pm

No I dont, btw where is this parameter? /etc/elasticsearch/elasticsearch.yml?

maybe is enabled

FredF · January 4, 2017, 3:20pm

Hi,

You have to set this param in the elasticsearch.yml file
https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html

Take a look at the doc here
https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-delete-index.html
It's about action.destructive_requires_name param

Carmine_Fabrizio · January 4, 2017, 3:33pm

I've never configured, and there is not this param.
Thanks for the link