Lost all index suddenly


(Carmine Fabrizio) #1

Hi guys,
I do not know why and how, but on 31/12 I've lost all the index , 1TB of index.
I have 3 nodes and on all of them I have the same issue,
Any ideas? someone knows something?

Regards
Carmine


(Antonio Bonuccelli) #2

Do you have the elasticsearch cluster logs, in particular the master ones, covering that period?


(Carmine Fabrizio) #3

Yes I do, I have a cluster ELK+Redis


(Antonio Bonuccelli) #4

Carmine, I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name).
This would be the first place to check to understand why these indices have disappeared.


(Carmine Fabrizio) #6

Yes understood, no I dont have a Elasticsearch cluster


(Antonio Bonuccelli) #7

Yes I do, I have a cluster ELK+Redis
Yes understood, no I dont have a Elasticsearch cluster

you do or you don't?


(Carmine Fabrizio) #8

Yo man, you're confusing me :joy:, I have 4 nodes configured like a cluster, I dont understand what do you mean here

I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name).

Thanks in advantage


(Antonio Bonuccelli) #9

Have a read at the basics perhaps it will clear confusion

The Basics

Main Elasticsearch logs are written to ES_HOME/logs/[cluster_name].log file. For this file the default level is INFO, thus being sufficient for a rather moderate amount of information and, at the same time, not create a huge log file.

that is the file(s) you want to check.


(Carmine Fabrizio) #10

Yes, thanks a lot :grin:, I've seen it before, and this is my output

[2016-12-30 00:00:01,441][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2016-12-30 00:00:01,767][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:01,769][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic)
[2016-12-30 00:00:02,969][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:04,047][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:53,727][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:01:21,381][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [mysql-error] (dynamic)
[2016-12-30 03:20:18,247][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 03:30:02,601][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.27] deleting index
[2016-12-30 08:38:10,445][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 09:48:35,035][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [jetty] (dynamic)
[2016-12-30 10:48:20,159][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic)
[2016-12-31 00:00:00,618][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2016-12-31 00:00:00,814][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:00:00,815][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic)
[2016-12-31 00:00:02,952][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:00:12,350][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:01:21,075][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [mysql-error] (dynamic)
[2016-12-31 00:06:43,654][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 02:39:51,829][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 03:30:03,397][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index
[2016-12-31 04:23:15,799][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [jetty] (dynamic)
[2016-12-31 06:37:16,910][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 09:13:45,504][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 10:17:20,024][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic)
[2017-01-01 00:00:00,710][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2017-01-01 00:00:00,943][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:00:00,944][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic)
[2017-01-01 00:00:04,076][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:00:23,679][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:01:21,274][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [mysql-error] (dynamic)
[2017-01-01 00:06:43,980][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:37:09,127][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 03:30:02,888][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2016.12.29] deleting index
[2017-01-01 21:06:58,120][INFO ][cluster.metadata         ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic)

(Antonio Bonuccelli) #11

what this shows leads me to think you have curator or a cron-like job set up to delete indices everyday at 3:30 AM


(Carmine Fabrizio) #12

I dont have anything, btw I've lost the index at 01:00am see the picture


(Antonio Bonuccelli) #13

I dont have anything

I take that means you don't have any cron job set up and I'd suggest to double check that.
The logs show that a DELETE is being called by someone, you could track this down using security auditing.
There is no auto-functionality to eliminate indices within Elasticsearch.


(Jörg Prante) #14

What timezone is this? CET? If so, was it the leap second?


(Carmine Fabrizio) #15

yes, I have curator configured

curator --host #IP delete indices --older-than 180 --time-unit days --timestring '%Y.%m.%d'

that's it


(Carmine Fabrizio) #16

no UTC

Wed Jan 4 14:30:31 UTC 2017

What do you mean?


(Antonio Bonuccelli) #17

yes, I have curator configured

I see, however looking at this line for example

[2016-12-31 03:30:03,397][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index

I'd assume curator is configured to delete indices older than 3 days , not 180...


#18

Do you enable "action.destructive_requires_name"?
It prevent removing indices through regex / "*"


(Carmine Fabrizio) #19

No I dont, btw where is this parameter? /etc/elasticsearch/elasticsearch.yml?

maybe is enabled


(Frederic Fabre) #20

Hi,

You have to set this param in the elasticsearch.yml file
https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html

Take a look at the doc here
https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-delete-index.html
It's about action.destructive_requires_name param


(Carmine Fabrizio) #21

I've never configured, and there is not this param.
Thanks for the link