# Lost all index suddenly **URL:** \ **Category:** Elasticsearch\ **Created:** [January 3, 2017, 11:20am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422 "2017-01-03T11:20:31Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 3, 2017, 11:20am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/1 "2017-01-03T11:20:31Z")
Hi guys, I do not know why and how, but on 31/12 I've lost all the index , 1TB of index. I have 3 nodes and on all of them I have the same issue, Any ideas? someone knows something? Regards Carmine ---
**Author:** ![nellicus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nellicus/32/51566_2.png) [@nellicus](https://discuss.elastic.co/u/nellicus)\ **Post date:** [January 3, 2017, 11:22am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/2 "2017-01-03T11:22:47Z")
Do you have the elasticsearch cluster logs, in particular the master ones, covering that period? ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 3, 2017, 11:25am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/3 "2017-01-03T11:25:22Z")
Yes I do, I have a cluster ELK+Redis ---
**Author:** ![nellicus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nellicus/32/51566_2.png) [@nellicus](https://discuss.elastic.co/u/nellicus)\ **Post date:** [January 3, 2017, 11:27am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/4 "2017-01-03T11:27:35Z")
Carmine, I am referring to Elasticsearch Cluster **Logs** (each node produces a log named after the cluster name). This would be the first place to check to understand why these indices have disappeared. ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 3, 2017, 11:34am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/6 "2017-01-03T11:34:40Z")
Yes understood, no I dont have a Elasticsearch cluster ---
**Author:** ![nellicus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nellicus/32/51566_2.png) [@nellicus](https://discuss.elastic.co/u/nellicus)\ **Post date:** [January 3, 2017, 11:35am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/7 "2017-01-03T11:35:26Z")
> Yes I do, I have a cluster ELK+Redis > Yes understood, no I dont have a Elasticsearch cluster you do or you don't? ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 3, 2017, 11:48am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/8 "2017-01-03T11:48:49Z")
Yo man, you're confusing me 😂, I have 4 nodes configured like a cluster, I dont understand what do you mean here > I am referring to Elasticsearch Cluster Logs (each node produces a log named after the cluster name). Thanks in advantage ---
**Author:** ![nellicus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nellicus/32/51566_2.png) [@nellicus](https://discuss.elastic.co/u/nellicus)\ **Post date:** [January 3, 2017, 11:52am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/9 "2017-01-03T11:52:27Z")
Have a read at the basics perhaps it will clear confusion > **[Elasticsearch Logging Secrets](https://www.elastic.co/blog/elasticsearch-logging-secrets)** > > Tips for Elasticsearch logging: logs inside Groovy scripts, custom loggers, dynamically changing the root logger, index level slowlog settings, deprecation logging. > The Basics > Main Elasticsearch logs are written to **ES\_HOME/logs/[cluster\_name].log file**. For this file the default level is INFO, thus being sufficient for a rather moderate amount of information and, at the same time, not create a huge log file. that is the file(s) you want to check. ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 3, 2017, 12:07pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/10 "2017-01-03T12:07:03Z")
Yes, thanks a lot 😁, I've seen it before, and this is my output ``` [2016-12-30 00:00:01,441][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog] [2016-12-30 00:00:01,767][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic) [2016-12-30 00:00:01,769][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic) [2016-12-30 00:00:02,969][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic) [2016-12-30 00:00:04,047][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic) [2016-12-30 00:00:53,727][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic) [2016-12-30 00:01:21,381][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [mysql-error] (dynamic) [2016-12-30 03:20:18,247][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic) [2016-12-30 03:30:02,601][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.27] deleting index [2016-12-30 08:38:10,445][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [syslog] (dynamic) [2016-12-30 09:48:35,035][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [jetty] (dynamic) [2016-12-30 10:48:20,159][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.30] update_mapping [rhino] (dynamic) [2016-12-31 00:00:00,618][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog] [2016-12-31 00:00:00,814][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic) [2016-12-31 00:00:00,815][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic) [2016-12-31 00:00:02,952][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic) [2016-12-31 00:00:12,350][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic) [2016-12-31 00:01:21,075][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [mysql-error] (dynamic) [2016-12-31 00:06:43,654][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic) [2016-12-31 02:39:51,829][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic) [2016-12-31 03:30:03,397][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index [2016-12-31 04:23:15,799][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [jetty] (dynamic) [2016-12-31 06:37:16,910][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic) [2016-12-31 09:13:45,504][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [syslog] (dynamic) [2016-12-31 10:17:20,024][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.31] update_mapping [rhino] (dynamic) [2017-01-01 00:00:00,710][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_, rhino, syslog] [2017-01-01 00:00:00,943][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic) [2017-01-01 00:00:00,944][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic) [2017-01-01 00:00:04,076][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic) [2017-01-01 00:00:23,679][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic) [2017-01-01 00:01:21,274][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [mysql-error] (dynamic) [2017-01-01 00:06:43,980][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic) [2017-01-01 00:37:09,127][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [syslog] (dynamic) [2017-01-01 03:30:02,888][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.29] deleting index [2017-01-01 21:06:58,120][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2017.01.01] update_mapping [rhino] (dynamic) ``` ---
**Author:** ![nellicus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nellicus/32/51566_2.png) [@nellicus](https://discuss.elastic.co/u/nellicus)\ **Post date:** [January 3, 2017, 12:40pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/11 "2017-01-03T12:40:28Z")
what this shows leads me to think you have curator or a cron-like job set up to delete indices everyday at 3:30 AM ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 3, 2017, 1:44pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/12 "2017-01-03T13:44:34Z")
I dont have anything, btw I've lost the index at 01:00am see the picture ![](https://us1.discourse-cdn.com/elastic/original/2X/2/22b1982fca0e0283c2248d27d5812b028d2e45b0.png) ---
**Author:** ![nellicus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nellicus/32/51566_2.png) [@nellicus](https://discuss.elastic.co/u/nellicus)\ **Post date:** [January 3, 2017, 1:58pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/13 "2017-01-03T13:58:01Z")
> I dont have anything I take that means you don't have any cron job set up and I'd suggest to double check that. The logs show that a DELETE is being called by someone, you could track this down using security auditing. There is no auto-functionality to eliminate indices within Elasticsearch. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 4, 2017, 1:24am UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/14 "2017-01-04T01:24:52Z")
What timezone is this? CET? If so, was it the leap second? ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 4, 2017, 2:30pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/15 "2017-01-04T14:30:20Z")
yes, I have curator configured > curator --host #IP delete indices --older-than 180 --time-unit days --timestring '%Y.%m.%d' that's it ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 4, 2017, 2:32pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/16 "2017-01-04T14:32:16Z")
no UTC > Wed Jan 4 14:30:31 UTC 2017 What do you mean? ---
**Author:** ![nellicus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nellicus/32/51566_2.png) [@nellicus](https://discuss.elastic.co/u/nellicus)\ **Post date:** [January 4, 2017, 2:36pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/17 "2017-01-04T14:36:48Z")
> yes, I have curator configured I see, however looking at this line for example > [2016-12-31 03:30:03,397][INFO][cluster.metadata] [leo-mt-r-elasticsearch02] [logstash-2016.12.28] deleting index I'd assume curator is configured to delete indices older than 3 days , not 180... ---
**Author:** ![junior\_h](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@junior\_h](https://discuss.elastic.co/u/junior_h)\ **Post date:** [January 4, 2017, 2:38pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/18 "2017-01-04T14:38:49Z")
Do you enable "action.destructive\_requires\_name"? It prevent removing indices through regex / "\*" ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 4, 2017, 3:08pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/19 "2017-01-04T15:08:39Z")
No I dont, btw where is this parameter? /etc/elasticsearch/elasticsearch.yml? maybe is enabled ---
**Author:** ![FredF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fredf/32/9957_2.png) [@FredF](https://discuss.elastic.co/u/FredF)\ **Post date:** [January 4, 2017, 3:20pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/20 "2017-01-04T15:20:27Z")
Hi, You have to set this param in the elasticsearch.yml file [https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html) Take a look at the doc here [https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-delete-index.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-delete-index.html) It's about action.destructive\_requires\_name param ---
**Author:** ![Carmine\_Fabrizio](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Carmine\_Fabrizio](https://discuss.elastic.co/u/Carmine_Fabrizio)\ **Post date:** [January 4, 2017, 3:33pm UTC](https://discuss.elastic.co/t/lost-all-index-suddenly/70422/21 "2017-01-04T15:33:51Z")
I've never configured, and there is not this param. Thanks for the link [Next page](https://discuss.elastic.co/t/lost-all-index-suddenly/70422.md?page=2)