Weekly index cleanup using curator 5.8

Elastic Stack Elasticsearch
1

I am trying to clean weekly indexes older than 4 weeks on elasticsearch 7.x however not able to figure out the curator config

Current config is

   -  filtertype: period
      period_type: relative
      source: name 
      range_from: -4
      range_to: 0
      timestring: '%Y.%m.%d'
      unit: weeks
      week_starts_on: monday

Per my understanding ( Correct if i am wrong ), this should target 4 weeks of worth index backdated from today.

I have an index for testing which i believe should be removed however its not being taken as part of cleanup criteria
2021-03-16 10:15:03,326 DEBUG curator.indexlist filter_by_age:550 Index "inf_os_logs-2021.11" does not meet provided criteria. Removing from list.

Could anyone suggest the right way to remove index older than 4 weeks with the following naming convention inf_XXXXXXX_logs-YYYY-WW as the template.

2

Your timestring looks for a date pattern, not a week. I suspect it should be %Y.%W or similar

3

Thank you , seems to be working. I am testing it out.

4

Please note that if using ISO weeks, your timestring will look like %G-%V instead. Python's datetime doesn't do ISO weeks without some extra calculations. A community contributor to Python added these and some extra code to compensate for ISO weeks.

2 Likes
5

While testing this out i have found that the curator is targeting indices currently in the 4 weeks as well

2021-03-18 06:05:27,974 DEBUG curator.indexlist __excludify:58 Remains in actionable list: Index "ams_fip-2021.02.25" age (1610323200), direction: "older", point of reference, (1613642727)

The age of the index seems to be different from the one shown in the elasticsearch and this is causing it to include index which are still needed

image

Date calculated by curator
GMT: Monday, January 11, 2021 12:00:00 AM
Your time zone: Monday, January 11, 2021 5:30:00 AM GMT+05:30
Relative: 2 months ago

Date shown on index
GMT: Thursday, February 25, 2021 12:00:02.031 AM
Your time zone: Thursday, February 25, 2021 5:30:02.031 AM GMT+05:30
Relative: 21 days ago

Am i missing anything ?

6

That is a daily index and I guess the month is interpreted as week number? You need to set prefixes and filters so you only catch weekly indices with a weekly delete step.

7

This is the current config, i have created 2 actions ( one for daily and other for weekly ). It appears the weekly is recognising the correct time frame daily is not.

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Test
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: '.*'
      exclude:
    - filtertype: creation_date
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: seconds
      unit_count: 2678400
      exclude:
actions:
  2:
    action: delete_indices
    description: >-
      Test-2
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: '.*'
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%W'
      unit: weeks
      unit_count: 4
      exclude:
8

@Christian_Dahlqvist is correct. The timestring patterns only get translated (at first) into digit matching regular expressions. %Y.%m.%d turns into \d{4}\.\d{2}\.\d{2} and %Y.%W turns into \d{4}\.\d{2}. Since the 2021.02 of 2021.02.25 matches both regular expressions, it is attempting to evaluate both.

To get around this, as suggested, you need to add a filter to omit year.month.day indices from your weekly index action.

  2:
    action: delete_indices
    description: >-
      Test-2
    options:
      ignore_empty_list: True
    filters:
    - filtertype: pattern
      kind: regex
      value: '^.*\d{4}\.\d{2}\.\{d}2$'
      exclude: true
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%W'
      unit: weeks
      unit_count: 4

The first filter in this example will exclude indices that end with a year.month.day pattern (i.e. 4 digits, a period, 2 digits, a period, and 2 more digits).

As an aside:

  1. You do not need to repeat actions for each action. Just preserve the indent level.
  2. You do not need to have an all inclusive pattern filter to catch everything .*. This is implied. All indices are included until filtered out (at least in the present version of Curator).
2 Likes
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.