Plugin to delete index data after some time interval


(Usha Datt) #1

I want to delete my index's data which exceeds time limit of 30 days, not the index. Is there any plugin available for it?
or if not, how to do the same?


(Mark Walkom) #2

Yep, check out https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html


(Usha Datt) #3

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

Delete all indices with matching --timestring which are older than 30 days

curator --host 10.0.0.2 alias --name lastmonth indices --newer-than 60 \
--older-than 30 --timestring '%Y.%m.%d' --time-unit days --prefix logstash

It specifies deletion of indexes which are older than 30 days, but I want to delete the data within a specified index which is older than 30 days. How to proceed with that?


(Jimferenczi) #4

You can check this page https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-ttl-field.html
Though it might not be a good idea to use a deprecated feature, you should probably try the delete by query plugin (https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugins-delete-by-query.html).


(Mark Walkom) #5

Don't use TTL!
Also don't use DBQ!

If you have data that is retained according to a time window, then use time based indices.


(Usha Datt) #6

Thanks @Warkolm
Yes , I was thinking the same to create the time based indices . But the issue is if I want to delete data after 30 days , I will have to create different indexes on per day basis.
So for one application I will be having 30 indexes , and will delete the indexes whose time-span is more than 30 days.
But when these application numbers is increased, creating 30 indexes/application , would not be a feasible solution.
So How should I proceed to use the time based indices ?


(Mark Walkom) #7

You can still do that, just reduce the shard count for smaller datasets.


(system) #8