Curator: Delete oldest indices based on ES cluster size

Elastic Stack Elasticsearch
2

An action file like this should suffice (if all indices match a given prefix):

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices matching the prefix MY_INDEX_PREFIX_HERE in excess of 
      25.5TB of data, starting with the oldest indices, based on index creation_date. 
      An empty index list (from no indices being in excess of the size limit, for 
      example) will not generate an error.
    options:
      ignore_empty_list: True
      timeout_override: 300
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: MY_INDEX_PREFIX_HERE
    - filtertype: space
      disk_space: 25500
      use_age: True
      source: creation_date

I suggest running Curator with the --dry-run flag to see the output before running it against your production database. This set of filters will first filter by index prefix (this will prevent accidentally deleting your .kibana index, for example), and then filter by the amount of disk space consumed, sorted by age (using the index creation_date as the time stamp), so the oldest indices are deleted first. You could also use source: name, if your indices have a date stamp in the index name. The documentation for the space filtertype should help you select a different means of determining age, if desired.

1 Like