Delete old data from Elasticsearch

(Siddharth ) #1


How to delete elasticsearch data which is older than 30 days from an Index. I have an Index and data keep on coming on daily basis , my requirement is to delete old data from this index to make more disk space .

What is the best option to achieve this?

(Mark Walkom) #2

The best option is to use time based indices, then you can simply delete the index with Elasticsearch Curator.

Without that you need to use delete-by-query, which is expensive.

(Srinivas Mangipudi) #3

Hi Mark,

Say I create monthly time-based indices, and delete the last month's index using curator. But when my index will switch to the latest month, I will not have any data on the first day. Is there some way or architecting that whenever my index switches I will atleast have 7 days worth of data to start with.

One way I am assuming could be duplicating data and start writing to both indices before 7 days the current index is about to expire.

Are there better ways to do this? Thanks.

(Mark Walkom) #4

Use weekly indices then.

(Christian Dahlqvist) #5

Create daily indices and every day drop the index which has aged beyond 30 days.

(Aaron Mildenstein) #6

Curator does not have to use a time stamp in the index name. It can use the creation_date or deterministically by testing the min or max time stamp values in the indices with the field_stats API. You could easily write filters for Curator to keep monthly index data until 7 days after a new month rolls over.

(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.