Curator - Delete oldest index

Elastic Stack Elasticsearch
1

Hi,

Is there a way to identify the oldest index and delete that using curator?
Basically, we want to rollover a index when they meet a size of 10GB and when we get 45 indices in total, we want to remove the oldest index.

Does curator support this behaviour? We cannot utilize ILM since we are using the OSS version.

2

Yes. Check out the count filter

3

Thanks.

Is there a way to identify the oldest index and always delete that, instead of relying on a static count?

4

Well, you first asked specifically about keeping 45 total indices, which the count filter does, and does by age by default. If there are 46 indices, the oldest one would be eliminated, keeping 45. If there are 50 indices, then it will delete 46 - 50, sorting and eliminating the oldest.

Otherwise, no. Curator can’t identify exactly and only the single oldest index with its current feature set.

5

Hi,

Yes correct sorry about that, not my intention to confuse things :slight_smile:

I noticed this: https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/fe_use_age.html

Given the curator count filter with the "use_age" option, consider this scenario:

  • We receive syslog and beats traffic
  • We save each to a separate index, logstash-2019.10-000001 & beats-2019.10-000001
  • Beats produce 3x the amount of logs that logstash does

If we simply say to keep 40 indices in the cluster there will be an uneven representation of beats and logstash logs, since we want to keep logstash for a longer period since those logs are so few. Would there be an obvious way to resolve this that i somehow have overlooked?

6

Sorry for the delayed response.

You can use the age filter (separate from or chained after the count filter) to exclude indices until they are outside a given date range. Even with rollover indices, you can do this by way of the field_stats functionality of the age filter to determine the minimum and/or maximum age of documents within the index. This means that you can instruct Curator to not perform an action unless even the newest document in an index is newer than n days.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.