How to automatically delete documents older than 3 months


I want to delete documents older than 3 months in my index. And I have read the doc here about using ILM to create time-based indices and apply ILM policies. However, this doesn't seem to fit our use case. We're using id we generate to write/update/read documents in the index. If we're to create time-based indices, we want to keep documents with a specific id unique across indices and this doesn't seem possible from what I learned.

Is there any other ways to automatically delete old documents that will fit my use case ?

Welcome to our community! :smiley:

No there's not, no. You will need to wrap something together with a delete by query.

So you keep some data for longer than 3 months?

Right now we have data older than 3 months. But we only need data within 3 months. For now we can do a manual delete-by-query, but in the future we want to automatically delete those old data. Is that possible ?

Yes, use time based indices and then delete those when they are older than 3 months.

But we also need to satisfy that each document should have a unique id across indices.

Ok, not sure why that doesn't mean you can't use time based indices though.

If that is the case time based indices may not be right for your use case and you will need to implement a way to trigger a periodic delete by query yourself as Elasticsearch does not support this.

For the trigger, I saw this watcher Getting started with Watcher | Elasticsearch Reference [7.11] | Elastic
Is watcher can be used to delete documents ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.