Deletion of indices action not enforced

Elastic Stack Elasticsearch
1

Hi Team,

I deployed EFK stack in my cluster with default settings except storage size and secrets settings. Everything is working so far. Now, I wanted to configure elasticsearch curator for managing indices in elasticsearch cluster. I already configured it with one action rule which deletes indices which are 90 days old. But the problem is, the action rule is not enforced and it is not deleting old indices from the cluster.

Here is the output of curator pod. Could anyone guide me what is missing in the rule.

2021-03-08 01:00:07,786 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2021-03-08 01:00:07,787 INFO                 curator.cli                    run:180  Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2021-03-08 01:00:07,787 INFO                 curator.cli                    run:201  Action ID: 1, "delete_indices" completed.
2021-03-08 01:00:07,787 INFO                 curator.cli                    run:202  Job completed.

I had gone through one github issue and it is being mentioned to use allow_ilm_indices: true in action.yaml. I made that change too. Still It did not help me. I did not configure any Index lifecycle management settings in Elasticsearch settings.

Here is my action.yaml file output.

action_file_yml: |-
    ---
    actions:
      1:
        action: delete_indices
        description: "Clean up ES by deleting old indices"
        options:
          timeout_override:
          continue_if_exception: False
          disable_action: False
          allow_ilm_indices: true
          ignore_empty_list: True
        filters:
        - filtertype: age
          source: name
          direction: older
          timestring: '%Y.%m.%d'
          unit: days
          unit_count: 90
          field:
          stats_result:
          epoch:
          exclude: False
10:27
 config_yml: |-
    ---
    client:
      hosts:
        - elasticsearch-master
      port: 9200
      url_prefix:
      use_ssl: True
      certificate: '/certs/elastic-stack-ca.pem'
      client_cert: '/certs/elastic-certificate.pem'
      ssl_no_validate: True
      http_auth: "XXXXX:XXXXXXXX"
      timeout: 30
      master_only: False
    logging:
      loglevel: DEBUG
      logfile:
      logformat: default
      blacklist: ['elasticsearch', 'urllib3']
2

This message simply implies that with all filtering having been done, no matching indices remain to act on. More of the debug log will be necessary to fully troubleshoot what happened.

3

Here is the output of cronjob pod which was triggered today. I deleted few of the index names, but the cluster has indexes which are 90 old.

2021-03-08 01:00:17,307 DEBUG          curator.indexlist        iterate_filters:1223 Un-parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 90, 'exclude': False, 'epoch': None, 'stats_result': 'min_value'}
2021-03-08 01:00:17,309 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7f4e5c9f6f28>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']")}
2021-03-08 01:00:17,309 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 90, 'exclude': False, 'epoch': None, 'stats_result': 'min_value'}
2021-03-08 01:00:17,309 DEBUG          curator.indexlist        iterate_filters:1230 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 90, 'exclude': False, 'epoch': None, 'stats_result': 'min_value'}
2021-03-08 01:00:17,309 DEBUG          curator.indexlist        iterate_filters:1239 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 90, 'exclude': False, 'epoch': None, 'stats_result': 'min_value'}
2021.01.07',  'kube-system-2021.02.26',  'kube-system-2021.01.10',  '.kibana_task_manager_1']
2021-03-08 01:00:17,309 DEBUG          curator.indexlist          filter_by_age:489  Filtering indices by age
2021-03-08 01:00:17,309 DEBUG          curator.indexlist   _get_name_based_ages:280  Getting ages of indices by "name"
2021-03-08 01:00:17,309 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2021-03-08 01:00:17,309 DEBUG              curator.utils         get_date_regex:200  regex = \d{4}\.\d{2}\.\d{2}
2021-03-08 01:00:17,310 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2021-03-08 01:00:17,317 DEBUG          curator.indexlist           working_list:237  Generating working list of indices

2021-03-08 01:00:17,317 DEBUG          curator.indexlist       __not_actionable:39   Index istio-system-2020.12.31 is not actionable, removing from list.
2021-03-08 01:00:17,317 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "istio-system-2020.12.31" age (1609372800), direction: "older", point of reference, (1607389217)

2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2021.03.01 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2021.03.01" age (1614556800), direction: "older", point of reference, (1607389217)
2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2021.03.08 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2021.03.08" age (1615161600), direction: "older", point of reference, (1607389217)
2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index istio-system-2021.02.09 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "istio-system-2021.02.09" age (1612828800), direction: "older", point of reference, (1607389217)

2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index istio-system-2021.02.04 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "istio-system-2021.02.04" age (1612396800), direction: "older", point of reference, (1607389217)

2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2020.12.28 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2020.12.28" age (1609113600), direction: "older", point of reference, (1607389217)
2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2021.01.26 is not actionable, removing from list.
2021-03-08 01:00:17,319 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2021.01.26" age (1611619200), direction: "older", point of reference, (1607389217)

2021.01.07',  'kube-system-2021.02.26',  'kube-system-2021.01.10',  '.kibana_task_manager_1']
2021-03-08 01:00:17,309 DEBUG          curator.indexlist          filter_by_age:489  Filtering indices by age
2021-03-08 01:00:17,309 DEBUG          curator.indexlist   _get_name_based_ages:280  Getting ages of indices by "name"
2021-03-08 01:00:17,309 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2021-03-08 01:00:17,309 DEBUG              curator.utils         get_date_regex:200  regex = \d{4}\.\d{2}\.\d{2}
2021-03-08 01:00:17,310 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2021-03-08 01:00:17,317 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2021-03-08 01:00:17,317 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2021.01.14 is not actionable, removing from list.
2021-03-08 01:00:17,317 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2021.01.14" age (1610582400), direction: "older", point of reference, (1607389217)

2021-03-08 01:00:17,317 DEBUG          curator.indexlist       __not_actionable:39   Index istio-system-2020.12.31 is not actionable, removing from list.
2021-03-08 01:00:17,317 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "istio-system-2020.12.31" age (1609372800), direction: "older", point of reference, (1607389217)

2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2021.03.01 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2021.03.01" age (1614556800), direction: "older", point of reference, (1607389217)
2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2021.03.08 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2021.03.08" age (1615161600), direction: "older", point of reference, (1607389217)
2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index istio-system-2021.02.09 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "istio-system-2021.02.09" age (1612828800), direction: "older", point of reference, (1607389217)

2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index istio-system-2021.02.04 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "istio-system-2021.02.04" age (1612396800), direction: "older", point of reference, (1607389217)

2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2020.12.28 is not actionable, removing from list.
2021-03-08 01:00:17,318 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2020.12.28" age (1609113600), direction: "older", point of reference, (1607389217)
2021-03-08 01:00:17,318 DEBUG          curator.indexlist       __not_actionable:39   Index kube-system-2021.01.26 is not actionable, removing from list.
2021-03-08 01:00:17,319 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "kube-system-2021.01.26" age (1611619200), direction: "older", point of reference, (1607389217)

2021-03-08 01:00:17,341 DEBUG          curator.indexlist        iterate_filters:1242 Post-instance: []
2021-03-08 01:00:17,341 DEBUG     curator.actions.delete_indices               __init__:477  master_timeout value: 30s
2021-03-08 01:00:17,341 DEBUG                curator.cli         process_action:99   Doing the action here.
2021-03-08 01:00:17,341 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2021-03-08 01:00:17,342 INFO                 curator.cli                    run:180  Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2021-03-08 01:00:17,342 INFO                 curator.cli                    run:201  Action ID: 1, "delete_indices" completed.
2021-03-08 01:00:17,342 INFO                 curator.cli                    run:202  Job completed.
/curator/lib/elasticsearch/connection/http_urllib3.py:178: UserWarning: Connecting to elasticsearch-master using SSL with verify_certs=False is insecure.
/curator/lib/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
/curator/lib/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
/curator/lib/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
/curator/lib/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. . See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

Thanks,

4

You have configured a unit_count of 90 unit's (days), and 90 days ago by my calculations is 8 December 2020. I do not see any indices that old in the output you shared, so it makes sense to me that no indices would be deleted.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.