Plugin to delete index data after some time interval

Elastic Stack Elasticsearch
1

I want to delete my index's data which exceeds time limit of 30 days, not the index. Is there any plugin available for it?
or if not, how to do the same?

2

Yep, check out https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

3

Delete all indices with matching --timestring which are older than 30 days

curator --host 10.0.0.2 alias --name lastmonth indices --newer-than 60 \
--older-than 30 --timestring '%Y.%m.%d' --time-unit days --prefix logstash

It specifies deletion of indexes which are older than 30 days, but I want to delete the data within a specified index which is older than 30 days. How to proceed with that?

4

You can check this page https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-ttl-field.html
Though it might not be a good idea to use a deprecated feature, you should probably try the delete by query plugin (https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugins-delete-by-query.html).

5

Don't use TTL!
Also don't use DBQ!

If you have data that is retained according to a time window, then use time based indices.

6

Thanks @Warkolm
Yes , I was thinking the same to create the time based indices . But the issue is if I want to delete data after 30 days , I will have to create different indexes on per day basis.
So for one application I will be having 30 indexes , and will delete the indexes whose time-span is more than 30 days.
But when these application numbers is increased, creating 30 indexes/application , would not be a feasible solution.
So How should I proceed to use the time based indices ?

7

You can still do that, just reduce the shard count for smaller datasets.