Delete old data from Elasticsearch

Elastic Stack Elasticsearch
1

Hi,

How to delete elasticsearch data which is older than 30 days from an Index. I have an Index and data keep on coming on daily basis , my requirement is to delete old data from this index to make more disk space .

What is the best option to achieve this?

2

The best option is to use time based indices, then you can simply delete the index with Elasticsearch Curator.

Without that you need to use delete-by-query, which is expensive.

3

Hi Mark,

Say I create monthly time-based indices, and delete the last month's index using curator. But when my index will switch to the latest month, I will not have any data on the first day. Is there some way or architecting that whenever my index switches I will atleast have 7 days worth of data to start with.

One way I am assuming could be duplicating data and start writing to both indices before 7 days the current index is about to expire.

Are there better ways to do this? Thanks.

4

Use weekly indices then.

5

Create daily indices and every day drop the index which has aged beyond 30 days.

1 Like
6

Curator does not have to use a time stamp in the index name. It can use the creation_date or deterministically by testing the min or max time stamp values in the indices with the field_stats API. You could easily write filters for Curator to keep monthly index data until 7 days after a new month rolls over.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.