How to automatically delete documents older than 3 months

Elastic Stack Elasticsearch
#1

Hey,

I want to delete documents older than 3 months in my index. And I have read the doc here about using ILM to create time-based indices and apply ILM policies. However, this doesn't seem to fit our use case. We're using id we generate to write/update/read documents in the index. If we're to create time-based indices, we want to keep documents with a specific id unique across indices and this doesn't seem possible from what I learned.

Is there any other ways to automatically delete old documents that will fit my use case ?

#2

Welcome to our community! :smiley:

No there's not, no. You will need to wrap something together with a delete by query.

So you keep some data for longer than 3 months?

#3

Right now we have data older than 3 months. But we only need data within 3 months. For now we can do a manual delete-by-query, but in the future we want to automatically delete those old data. Is that possible ?

#4

Yes, use time based indices and then delete those when they are older than 3 months.

#5

But we also need to satisfy that each document should have a unique id across indices.

#6

Ok, not sure why that doesn't mean you can't use time based indices though.

#7

If that is the case time based indices may not be right for your use case and you will need to implement a way to trigger a periodic delete by query yourself as Elasticsearch does not support this.

#8

For the trigger, I saw this watcher Getting started with Watcher | Elasticsearch Reference [7.11] | Elastic
Is watcher can be used to delete documents ?

#9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.