Index Automatically Deleted Every Few Days – ads_index Disappears Without ILM Enabled

Elastic Stack Elasticsearch
1

Hello

We’re encountering a critical issue where our custom index (ads_index) is automatically deleted every 3–4 days without any manual action or lifecycle management policies in effect.

Elasticsearch version: 8.18
Single-node instance
Index in question: ads_index
ILM status: disabled for this index

We’ve carefully reviewed the logs and noticed a recurring pattern. Every few days, the index ads_index is deleted automatically, along with another index read_me, and only the read_me index is recreated afterward — our index is not restored in any form.

Here are two such instances:

June 8, 2025
[2025-06-08T23:26:22,052][INFO ][o.e.c.m.MetadataDeleteIndexService] [demosite.com] [ads_index/vYcd1axoQhWSpxI94UW9sA] deleting index [2025-06-08T23:26:22,656][INFO ][o.e.c.m.MetadataDeleteIndexService] [demosite.com] [read_me/bLRUm6yOTZqRIpaQavX3-w] deleting index [2025-06-08T23:26:23,112][INFO ][o.e.c.m.MetadataCreateIndexService] [demosite.com] [read_me] creating index, cause [api], templates [], shards [1]/[1] [2025-06-08T23:26:23,531][INFO ][o.e.c.m.MetadataMappingService] [demosite.com] [read_me/H5OlWKfFTYu_6LDjr16x5g] create_mapping

June 14, 2025
[2025-06-14T22:03:01,803][INFO ][o.e.c.m.MetadataDeleteIndexService] [demosite.com] [ads_index/SMc_M8tkTV-mq8CtEUZJag] deleting index [2025-06-14T22:03:02,245][INFO ][o.e.c.m.MetadataDeleteIndexService] [demosite.com] [read_me/qjy4wXTyT4-D1B1HehE2NQ] deleting index [2025-06-14T22:03:02,588][INFO ][o.e.c.m.MetadataCreateIndexService] [demosite.com] [read_me] creating index, cause [api], templates [], shards [1]/[1] [2025-06-14T22:03:02,862][INFO ][o.e.c.m.MetadataMappingService] [demosite.com] [read_me/WRWCPDnzR9uGuQepFU3_lg] create_mapping

Later that same night:

[2025-06-14T23:01:52,927][INFO ][o.e.c.m.MetadataDeleteIndexService] [demosite.com] [read_me/WRWCPDnzR9uGuQepFU3_lg] deleting index [2025-06-14T23:01:53,037][INFO ][o.e.c.m.MetadataCreateIndexService] [demosite.com] [read_me] creating index, cause [api], templates [], shards [1]/[1] [2025-06-14T23:01:53,213][INFO ][o.e.c.m.MetadataMappingService] [demosite.com] [read_me/j4ww2hEfSm2TnoztJAXTBw] create_mapping
Our index (ads_index) is not recreated automatically. It just disappears silently every few days.

Troubleshooting We Did
Verified that ILM is disabled for ads_index using GET _ilm/explain:
"managed": false
We adjusted and even removed index templates to prevent deletion triggers.
No delete operations are performed from our side or our API clients.
We’ve tried setting index.blocks.write and index.blocks.delete, but the index still gets deleted.

Additionally, we’ve checked server status via systemd:

sudo systemctl status elasticsearch
Active: active (running) since Fri 2025-06-13 00:36:04
This uptime aligns with the moment the last deletion occurred, which hints that maybe a restart or crash is triggering the deletions.

Our Questions
Is there any internal Elastic mechanism or system component that could cause automatic deletion like this?
Could there be a background recovery task that only restores system-like indices (like read_me) but skips user indices?
Could cluster crash recovery or corrupted state during startup be a reason ads_index is not restored?
Is there a way to trace who/what is calling index deletion? We don't see any user action or API log beyond MetadataDeleteIndexService.

We would greatly appreciate any guidance or tools we can use to:

Identify the root cause of these automatic deletions;
Prevent further data loss;
Ensure ads_index is recreated or restored properly if a failure occurs.

I hope for your help, I have been struggling with this bug for almost a month.

Thank you in advance!

Best regards,
Arsen

2

Hi @Developer_G_Design

Welcome to the community

No there is no mechanism in elasticsearch that deletes indices on its own / without configuring ILM / DSL or some external script.

So my first question is is your elastic cluster secured with TLS and authentication? And or open to the Internet?

Second, if you're cluster is restarting/ crashing, you need to focus on that.

Perhaps share your setup

3

From everything you described it looks like that your cluster is exposed to the internet with security disabled.

This behavior is pretty common and the presence of an index named read_me is a strong indication that someone is deleting your data and writing this index with some information.

1 Like
4

Hi, @stephenb thank you for your response. To be honest, I didn't know that this could happen, i.e. do you think that if the port is open, there could be requests from outside? Is this some kind of automatic work?
We are just confused that the deletion works with a certain periodicity, could this happen from outside? Please share your experience.
P.S. The test site is currently closed to visitors.

5

Hi, @leandrojmp , yes, maybe you are right, I checked the elasticsearch.yml and I see that it is installed there - xpack.security.enabled: false
I just need to enable protection and that's it? Or are there any other actions?

6

@Developer_G_Design

My experience is that if you leave an elasticsearch cluster exposed on the internet without authentication that it will be compromised within minutes.

So you did not answer our questions. Did you have SSL and authentication enable? If not, you most likely had your cluster compromised from the outside.

You'll most likely need to start from scratch. Make sure your elasticsearch cluster is fully secured.

It is another possibility if some sort of script injection through your front end... We don't know what your friend didn't is and we really can't help you with that happens to be the case.

This is all normal secure application development stuff.

How did you set up the elasticsearch?

We can help with that. But we can't really provide complete application secure coding guideline.

7

@stephenb now I see it in the settings

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

But I also see that

xpack.security.enabled: false

Do I understand correctly that I just need to turn it(true) on and everything will work after server reload?
Or do I need to additionally create a password and connect using this password?

8

Yup unsecured... curious what documentation you were following that instructed you to do that... did you follow this? Note the warning....

At this point you should clean everything up and start over.

No just turning that to true will not fix everything... there would be other steps invovled.

My Recommendation : You need to start from scratch and build the single node properly.

If you simply install elasticsearch with everything default you will get a secured cluster...

Please follow one of the guides etc.

1 Like
9

@stephenb thanks for your help, I'll try to do everything, if I have any questions, I'll write

1 Like