Hi guys,
I do not know why and how, but on 31/12 I've lost all the index , 1TB of index.
I have 3 nodes and on all of them I have the same issue,
Any ideas? someone knows something?
Regards
Carmine
Hi guys,
I do not know why and how, but on 31/12 I've lost all the index , 1TB of index.
I have 3 nodes and on all of them I have the same issue,
Any ideas? someone knows something?
Regards
Carmine
Do you have the elasticsearch cluster logs, in particular the master ones, covering that period?
Yes I do, I have a cluster ELK+Redis
Carmine, I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name).
This would be the first place to check to understand why these indices have disappeared.
Yes understood, no I dont have a Elasticsearch cluster
Yes I do, I have a cluster ELK+Redis
Yes understood, no I dont have a Elasticsearch cluster
you do or you don't?
Yo man, you're confusing me , I have 4 nodes configured like a cluster, I dont understand what do you mean here
I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name).
Thanks in advantage
Have a read at the basics perhaps it will clear confusion
The Basics
Main Elasticsearch logs are written to ES_HOME/logs/[cluster_name].log file. For this file the default level is INFO, thus being sufficient for a rather moderate amount of information and, at the same time, not create a huge log file.
that is the file(s) you want to check.
Yes, thanks a lot , I've seen it before, and this is my output
[2016-12-30 00:00:01,441][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2016-12-30 00:00:01,767][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:01,769][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic)
[2016-12-30 00:00:02,969][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:04,047][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:00:53,727][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 00:01:21,381][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [mysql-error] (dynamic)
[2016-12-30 03:20:18,247][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 03:30:02,601][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.27] deleting index
[2016-12-30 08:38:10,445][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic)
[2016-12-30 09:48:35,035][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [jetty] (dynamic)
[2016-12-30 10:48:20,159][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic)
[2016-12-31 00:00:00,618][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2016-12-31 00:00:00,814][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:00:00,815][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic)
[2016-12-31 00:00:02,952][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:00:12,350][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 00:01:21,075][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [mysql-error] (dynamic)
[2016-12-31 00:06:43,654][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 02:39:51,829][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 03:30:03,397][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index
[2016-12-31 04:23:15,799][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [jetty] (dynamic)
[2016-12-31 06:37:16,910][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 09:13:45,504][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic)
[2016-12-31 10:17:20,024][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic)
[2017-01-01 00:00:00,710][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog]
[2017-01-01 00:00:00,943][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:00:00,944][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic)
[2017-01-01 00:00:04,076][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:00:23,679][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:01:21,274][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [mysql-error] (dynamic)
[2017-01-01 00:06:43,980][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 00:37:09,127][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic)
[2017-01-01 03:30:02,888][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.29] deleting index
[2017-01-01 21:06:58,120][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic)
what this shows leads me to think you have curator or a cron-like job set up to delete indices everyday at 3:30 AM
I dont have anything, btw I've lost the index at 01:00am see the picture
I dont have anything
I take that means you don't have any cron job set up and I'd suggest to double check that.
The logs show that a DELETE is being called by someone, you could track this down using security auditing.
There is no auto-functionality to eliminate indices within Elasticsearch.
What timezone is this? CET? If so, was it the leap second?
yes, I have curator configured
curator --host #IP delete indices --older-than 180 --time-unit days --timestring '%Y.%m.%d'
that's it
no UTC
Wed Jan 4 14:30:31 UTC 2017
What do you mean?
yes, I have curator configured
I see, however looking at this line for example
[2016-12-31 03:30:03,397][INFO ][cluster.metadata ] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index
I'd assume curator is configured to delete indices older than 3 days , not 180...
Do you enable "action.destructive_requires_name"?
It prevent removing indices through regex / "*"
No I dont, btw where is this parameter? /etc/elasticsearch/elasticsearch.yml?
maybe is enabled
Hi,
You have to set this param in the elasticsearch.yml file
https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html
Take a look at the doc here
https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-delete-index.html
It's about action.destructive_requires_name param
I've never configured, and there is not this param.
Thanks for the link
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.