# My ElasticSearch Indexes have been mysteriously deleted, how do I debug the cause? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 13, 2017, 2:19am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460 "2017-01-13T02:19:26Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![sameerpanjwani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sameerpanjwani/32/14530_2.png) [@sameerpanjwani](https://discuss.elastic.co/u/sameerpanjwani)\ **Post date:** [January 13, 2017, 2:19am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/1 "2017-01-13T02:19:26Z")
I just checked my ElasticSearch server and the indexes are gone, there is no index remaining. I don't see any information in the /var/logs/elasticsearch and not sure what to look for. 1. How do I debug what went wrong and how it got deleted? 2. My elasticsearch is accessible via a public IP on the 9200 port, I haven't done anything to secure it, can it be that? Also, how do I secure it especially when I need to use it myself to call from my own set of servers. Mostly trying to figure what went wrong and how to prevent it from happening again. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 13, 2017, 2:27am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/2 "2017-01-13T02:27:31Z")
Exposing things to the internet without any sort of security is a Bad Thing. See [Ransom attack on Elasticsearch cluster?](https://discuss.elastic.co/t/ransom-attack-on-elasticsearch-cluster/71310/) and then [https://www.elastic.co/products/x-pack/security](https://www.elastic.co/products/x-pack/security) ---
**Author:** ![sameerpanjwani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sameerpanjwani/32/14530_2.png) [@sameerpanjwani](https://discuss.elastic.co/u/sameerpanjwani)\ **Post date:** [January 13, 2017, 2:50am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/3 "2017-01-13T02:50:12Z")
Does Elastic log delete index requests anywhere? Like when it happened, and if so, from which IP? I know I need to secure things and working on that, but still would be useful to know when it happened and which IP triggered it. Is there any log for that? ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [January 13, 2017, 8:58am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/4 "2017-01-13T08:58:48Z")
Not by default, you will need to install an audit-log plugin to do that. Also see [http://code972.com/blog/2017/01/107-dont-be-ransacked-securing-your-elasticsearch-cluster-properly](http://code972.com/blog/2017/01/107-dont-be-ransacked-securing-your-elasticsearch-cluster-properly) for instructions on how to protect your cluster going forward. ---
**Author:** ![switchedfabric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/switchedfabric/32/14508_2.png) [@switchedfabric](https://discuss.elastic.co/u/switchedfabric)\ **Post date:** [January 13, 2017, 1:07pm UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/5 "2017-01-13T13:07:33Z")
I had the same. > [@Indices all gone but i don't think i did it](https://discuss.elastic.co/t/indices-all-gone-but-i-dont-think-i-did-it/71370): > > Hi all, I've been running ES 5 for some time (say a month) now and all has been well, but last night all my indices were deleted. I see this in the log (the second line): ---- 8\< ---- [2017-01-11T21:23:04,357][WARN][o.e.d.z.p.PublishClusterStateAction] [master] timed out waiting for all nodes to process published state ... [2017-01-11T21:23:08,473][WARN][o.e.c.s.ClusterService] [master] cluster state update task [delete-index ...] took [34.1s] above the warn threshold of 30s ---- 8\< … But no index with a ransom note :-. ---
**Author:** ![cmcc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmcc/32/7180_2.png) [@cmcc](https://discuss.elastic.co/u/cmcc)\ **Post date:** [January 16, 2017, 3:07pm UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/6 "2017-01-16T15:07:07Z")
I have had the same thing happen 2 or 3 times over the past few days, twice alone today. Using version 5.1.1 Additional ES plugins used: discovery-ec2, ingest-attachment, repository-s3 First time preceded by a `ResourceLeakDetector` ```auto e[36melasticsearch_1 |e[0m [2017-01-15T15:36:12,423][ERROR][i.n.u.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information. e[36melasticsearch_1 |e[0m [2017-01-16T04:16:00,784][WARN][o.e.c.s.ClusterService] [fFMWjXz] cluster state update task [delete-index [[workflow_private0/3t6AaGLsRXi5-jcEYH48Tg], [infrastructure0/oPjqaDXnRN27Lx-e8yEzMw], [content_private0/vk-NPOsTRdSJZjNbyQn6BA], [management/T2OllFSgRv6HPD6LaDfBCQ], [cluster/85CYxbVDQYm6fuYkO2PoGA], [content_public0/PV3oTLY4Q36FMjPKuOhSsw], [workflow_public0/nTIzxcGMQ3a0ifWOe4ta0w], [cpriv_htb/RESMZPU2T9ahDB0bbAE4Sg], [.kibana/3Kqif065T0ii9EpRpYSXjw]]] took [1.4m] above the warn threshold of 30s ``` Second time ```auto e[36melasticsearch_1 |e[0m [2017-01-16T14:27:53,432][WARN][o.e.c.s.ClusterService] [fFMWjXz] cluster state update task [delete-index [[content_public0/D1dBk1ZbToifU2XHCK9eeQ], [workflow_public0/NW_1-4qiSFi_8UW3UhNdhQ], [infrastructure0/WX1RIWKDS3uKlvgSvEDp8w], [workflow_private0/HM0Ks0xKRdWLhnmPrlCF3g], [management/vp0BgpKGRFKmgzoH38f3Fw], [content_private0/A2JshrHYR1e2Vc8Mw0W_vg], [.kibana/Kuesne1vR0mLb-NMwSDHgg], [cluster/88Iwr1-bTPSr-efMK2g7Ig]]] took [1.4m] above the warn threshold of 30s ``` Previously this test-environment (admittedly exposed to the internet) had been running fine for months. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 16, 2017, 3:58pm UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/7 "2017-01-16T15:58:06Z")
About the first message it has been fixed AFAIK in 5.1.2. The second problem is related to this I think: > [@Ransom attack on Elasticsearch cluster?](https://discuss.elastic.co/t/ransom-attack-on-elasticsearch-cluster/71310): > > Today I found that all indices on our Test ES cluster was removed and one new index "warning" was created there. And I found following text from the raw index data: SEND 0.2 BTC TO THIS WALLET: 1DAsGY4Kt1a4LCTPMH5vm5PqX32eZmot4r IF YOU WANT RECOVER YOUR DATABASE! SEND TO THIS EMAIL YOUR SERVER IP AFTER SENDING THE BITCOINS... It is a tipical ransom attack on MongoDB recently: [https://www.bleepingcomputer.com/news/security/number-of-hijacked-mongodb-databases-is-going-up-as-more-hackers-are-f…](https://www.bleepingcomputer.com/news/security/number-of-hijacked-mongodb-databases-is-going-up-as-more-hackers-are-flocking-in/) ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 17, 2017, 11:34am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/8 "2017-01-17T11:34:24Z")
> [@dadoonet](#): > > About the first message it has been fixed AFAIK in 5.1.2. No, this appears to be a different and far more serious problem. ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 17, 2017, 11:35am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/9 "2017-01-17T11:35:19Z")
> [@cmcc](#): > > ```auto > [2017-01-15T15:36:12,423][ERROR][i.n.u.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information. > > ``` Can you provide any details about what was happening on the cluster before you saw this message? ---
**Author:** ![cmcc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmcc/32/7180_2.png) [@cmcc](https://discuss.elastic.co/u/cmcc)\ **Post date:** [January 17, 2017, 4:26pm UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/10 "2017-01-17T16:26:48Z")
So I had restored a indexes from a 5.0.0-alpha5 s3-repository using that 'restore' function, into an ES instance running 5.1.1. I can send you logs if you give me an email address. ---
**Author:** ![vamshi](https://avatars.discourse-cdn.com/v4/letter/v/ecccb3/32.png) [@vamshi](https://discuss.elastic.co/u/vamshi)\ **Post date:** [January 20, 2017, 3:49am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/11 "2017-01-20T03:49:31Z")
Mostly unsecured elastic servers/public ones are never guaranteed. There can any reason the person with host address can simply delete the indexes. So i suggest, instead of worrying about something after it went wrong secure your cluster of nodes. Use X-pack introduced for latest elastic version, It had shield to secure & Marvel as head [https://www.elastic.co/downloads/x-pack](https://www.elastic.co/downloads/x-pack) Also I suggest you to have a backup folder with your indexes. You use script to take backups. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 17, 2017, 3:49am UTC](https://discuss.elastic.co/t/my-elasticsearch-indexes-have-been-mysteriously-deleted-how-do-i-debug-the-cause/71460/12 "2017-02-17T03:49:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.