Indices deleting using curator (different retention period for different index)

Elastic Stack Elasticsearch
1

hi All,
I have a question regarding deleting indices.

say i have many index data below, i want to purge A-YYYY.MM.DD when > 14 days, but for rest of the indexes B, C ...Z purge when > 7 days. How can I do that?
A-YYYY.MM.DD
B-YYYY.MM.DD
C-YYYY.MM.DD
...
Z-YYYY.MM.DD

2

By using multiple actions, each with different prefix filters and age filters:

---
actions:
  1:
    action: delete_indices
    ...
    filters:
    - filtertype: pattern
      kind: prefix
      value: A-
    - filtertype: pattern
      kind: timestring
      value: %Y.%m.%d
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 14
  2:
    action: delete_indices
    ...
    filters:
    - filtertype: pattern
      kind: regex
      value: '^B-|^C-|^D-'
    - filtertype: pattern
      kind: timestring
      value: %Y.%m.%d
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7

This is a hypothetical action file with two actions. The first would delete indices that start with A- and contain a YEAR.MONTH.DAY timestring, after they were determined to be older than 14 days (based on the index creation date). The second would delete any indices starting with B-, C-, or D-, that also contain a YEAR.MONTH.DAY timestring, after they were determined to be older than 7 days (based on the index creation date). No other configuration data is included here, so test thoroughly with the --dry-run flag before actually running something against your production data.

3

hi Thanks for your reply.
so the idea is we dont want to setup based on eachindex like "value: '^B-|^C-|^D-'" because everytime we have to add in if having a new index.

so probably i can put the 2nd action as below, does it work?
2:
action: delete_indices
...
filters:
- filtertype: pattern
kind: regex
value: A-
exclude: True
- filtertype: pattern
kind: timestring
value: %Y.%m.%d
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 7

4

This will partly work the way you expect, in that it will get all indices except the ones that start with A-. However, what if there are other indices in your system? Do you use Kibana? There will be .kibana-type indices that would still be seen by this, as they do not start with A-.

You might still need additional filters to achieve exactly what you want, but you are on the right track. Use the --dry-run flag and check your logs to see for sure before committing to a course of action in production.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.