My ElasticSearch Indexes have been mysteriously deleted, how do I debug the cause?

[sameerpanjwani (Sameer)]

I just checked my ElasticSearch server and the indexes are gone, there is no index remaining. I don't see any information in the /var/logs/elasticsearch and not sure what to look for.

1. How do I debug what went wrong and how it got deleted?

2. My elasticsearch is accessible via a public IP on the 9200 port, I haven't done anything to secure it, can it be that? Also, how do I secure it especially when I need to use it myself to call from my own set of servers.

Mostly trying to figure what went wrong and how to prevent it from happening again.

[warkolm (Mark Walkom)]

Exposing things to the internet without any sort of security is a Bad Thing.

See Ransom attack on Elasticsearch cluster? and then https://www.elastic.co/products/x-pack/security

[sameerpanjwani (Sameer)]

Does Elastic log delete index requests anywhere? Like when it happened, and if so, from which IP? I know I need to secure things and working on that, but still would be useful to know when it happened and which IP triggered it. Is there any log for that?

[Itamar_Syn_Hershko]

Not by default, you will need to install an audit-log plugin to do that.

Also see http://code972.com/blog/2017/01/107-dont-be-ransacked-securing-your-elasticsearch-cluster-properly for instructions on how to protect your cluster going forward.

[switchedfabric (Willem Roos)]

I had the same.

Indices all gone but i don't think i did it

Hi all, I've been running ES 5 for some time (say a month) now and all has been well, but last night all my indices were deleted. I see this in the log (the second line): ---- 8< ---- [2017-01-11T21:23:04,357][WARN ][o.e.d.z.p.PublishClusterStateAction] [master] timed out waiting for all nodes to process published state ... [2017-01-11T21:23:08,473][WARN ][o.e.c.s.ClusterService ] [master] cluster state update task [delete-index ... ] took [34.1s] above the warn threshold of 30s ---- 8< …

But no index with a ransom note :-.

[cmcc]

I have had the same thing happen 2 or 3 times over the past few days, twice alone today.
Using version 5.1.1
Additional ES plugins used: discovery-ec2, ingest-attachment, repository-s3

First time preceded by a ResourceLeakDetector

e[36melasticsearch_1  |e[0m [2017-01-15T15:36:12,423][ERROR][i.n.u.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.
e[36melasticsearch_1  |e[0m [2017-01-16T04:16:00,784][WARN ][o.e.c.s.ClusterService   ] [fFMWjXz] cluster state update task [delete-index [[workflow_private0/3t6AaGLsRXi5-jcEYH48Tg], [infrastructure0/oPjqaDXnRN27Lx-e8yEzMw], [content_private0/vk-NPOsTRdSJZjNbyQn6BA], [management/T2OllFSgRv6HPD6LaDfBCQ], [cluster/85CYxbVDQYm6fuYkO2PoGA], [content_public0/PV3oTLY4Q36FMjPKuOhSsw], [workflow_public0/nTIzxcGMQ3a0ifWOe4ta0w], [cpriv_htb/RESMZPU2T9ahDB0bbAE4Sg], [.kibana/3Kqif065T0ii9EpRpYSXjw]]] took [1.4m] above the warn threshold of 30s

Second time

e[36melasticsearch_1  |e[0m [2017-01-16T14:27:53,432][WARN ][o.e.c.s.ClusterService   ] [fFMWjXz] cluster state update task [delete-index [[content_public0/D1dBk1ZbToifU2XHCK9eeQ], [workflow_public0/NW_1-4qiSFi_8UW3UhNdhQ], [infrastructure0/WX1RIWKDS3uKlvgSvEDp8w], [workflow_private0/HM0Ks0xKRdWLhnmPrlCF3g], [management/vp0BgpKGRFKmgzoH38f3Fw], [content_private0/A2JshrHYR1e2Vc8Mw0W_vg], [.kibana/Kuesne1vR0mLb-NMwSDHgg], [cluster/88Iwr1-bTPSr-efMK2g7Ig]]] took [1.4m] above the warn threshold of 30s

Previously this test-environment (admittedly exposed to the internet) had been running fine for months.

[dadoonet (David Pilato)]

About the first message it has been fixed AFAIK in 5.1.2.
The second problem is related to this I think:

Ransom attack on Elasticsearch cluster?

Today I found that all indices on our Test ES cluster was removed and one new index "warning" was created there. And I found following text from the raw index data: SEND 0.2 BTC TO THIS WALLET: 1DAsGY4Kt1a4LCTPMH5vm5PqX32eZmot4r IF YOU WANT RECOVER YOUR DATABASE! SEND TO THIS EMAIL YOUR SERVER IP AFTER SENDING THE BITCOINS... It is a tipical ransom attack on MongoDB recently: https://www.bleepingcomputer.com/news/security/number-of-hijacked-mongodb-databases-is-going-up-as-more-hackers-are-f…

[jasontedor]

@dadoonet

About the first message it has been fixed AFAIK in 5.1.2.

No, this appears to be a different and far more serious problem.

[jasontedor]

@cmcc

[2017-01-15T15:36:12,423][ERROR][i.n.u.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.

Can you provide any details about what was happening on the cluster before you saw this message?

[cmcc]

So I had restored a indexes from a 5.0.0-alpha5 s3-repository using that 'restore' function, into an ES instance running 5.1.1. I can send you logs if you give me an email address.

[vamshi (Vamshi Krishna Muppala)]

Mostly unsecured elastic servers/public ones are never guaranteed. There can any reason the person with host address can simply delete the indexes. So i suggest, instead of worrying about something after it went wrong secure your cluster of nodes.

Use X-pack introduced for latest elastic version, It had shield to secure & Marvel as head
https://www.elastic.co/downloads/x-pack

Also I suggest you to have a backup folder with your indexes. You use script to take backups.