ES curator not deleting the indices data

Elastic Stack Elasticsearch
1

Hello,

I have installed the ES-curator and below are my curator.yml and action.yml.

I am seeing below errors while running the dry run and also when I tried with the cronjob entries. Because I don't see the indices data for last 7 days are not getting deleted.

Please need suggestion.

curator.yml

client:
  hosts:
    - localhost
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:   '/home/ravi/.curator/log.log'
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

action.yml

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for filebeat-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-*
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7

Logs

2024-01-02 14:04:01,616 INFO      Creating client object and testing connection
2024-01-02 14:04:01,617 INFO      Instantiating client object
2024-01-02 14:04:01,617 INFO      Testing client connectivity
2024-01-02 14:04:01,623 INFO      Successfully created Elasticsearch client object with provided settings
2024-01-02 14:04:01,625 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 7 days (based on index name), for filebeat- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2024-01-02 14:04:02,048 INFO      **Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>**
2024-01-02 14:04:02,049 INFO      Action ID: 1, "delete_indices" completed.
2024-01-02 14:04:02,049 INFO      Job completed.

# curl -X GET "http://localhost:9200/_cat/indices?v"
health status index                                                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2023.12.12-000001                              tPWnGi4lTM6CtbsP0TIXHg   1   1        124            0    102.4kb        102.4kb
yellow open   logstash-202312_202312                                  hmKdhhQbRi-cEhUwfG5SbA   1   1       7066            0        8mb            8mb
green  open   .kibana_7.17.13_001                                     _sqv--IVRhqSxaCU-mwEoA   1   0       3652          333      3.2mb          3.2mb
yellow open   filebeat-7.17.13-2023.09.20                             -8k2a-qTQQyW9XrUktAYmA   1   1      20004            0        5mb            5mb
yellow open   filebeat-7.17.13-2023.09.21                             HzS4Ppc5QhKqDiyHWCJwWw   1   1     102737            0     16.7mb         16.7mb
yellow open   filebeat-7.17.13-2024.01.01-000005                      9UnVNkJAS1aaUXEdrt2zZA   1   1          0            0       227b           227b
yellow open   filebeat-7.17.13-2023.12.25-000004                      m34Z1tFLSeqDSEezkaVw3Q   1   1          0            0       227b           227b
green  open   .geoip_databases                                        pjntUuOETdCoosupsySPJg   1   0          4            0      2.9mb          2.9mb
yellow open   filebeat-7.17.13-2023.12.18                             xM2bX6wASWyAVgu3v4NG2w   1   1        287            0    188.2kb        188.2kb
yellow open   logstash-2023.12.12-000001_202312                       ayCVf3jWQYS0Anzi9aZzPQ   1   1         23            0     19.3kb         19.3kb
green  open   .apm-custom-link                                        IL--sRv-SgyhpLgzUwDLLQ   1   0          0            0       227b           227b
yellow open   filebeat-7.17.13-2023.12.15                             QGs-sawVQ3SGzQoWSyBEYw   1   1      92750            0     19.3mb         19.3mb
green  open   .kibana_task_manager_7.17.13_001                        2AS_qcRzSRyQlI7UgkrkVg   1   0         17        12615      1.7mb          1.7mb
yellow open   %{[@metadata][beat]}-%{[@metadata][version]}-2023.12.26 1BSodudrQZiOlLVnWdyQdQ   1   1       3236            0      3.7mb          3.7mb
yellow open   %{[@metadata][beat]}-%{[@metadata][version]}-2023.12.25 Ey0s0KYbQkaW8WMdPSoMGw   1   1       3821            0      4.8mb          4.8mb
green  open   .apm-agent-configuration                                GKpm77nJT8G9p7AXuolVNQ   1   0          0            0       227b           227b
yellow open   filebeat-7.17.13-2023.09.14                             iq6AyqSJTX-2jX12nzUh1A   1   1       3990            0      1.6mb          1.6mb
green  open   .tasks                                                  r4a_4MdATcCNKRTnORxdmA   1   0        124            5     90.4kb         90.4kb
yellow open   filebeat-7.17.13-2023.09.15                             dbp45fHcSXi37MLtnzMD-A   1   1       1560            0      1.3mb          1.3mb
yellow open   filebeat-7.17.13-2023.12.29                             bgI3JWqJQv-CF3URgOboXA   1   1     360230            0     40.7mb         40.7mb
yellow open   filebeat-7.17.13-2023.09.18                             9IBSh-OSQF2l2cX41vkF2A   1   1       4404            0      1.5mb          1.5mb
yellow open   filebeat-7.17.13-2023.12.28                             aN5saI_kRJmrXBTWhZ7aUg   1   1     157875            0     17.7mb         17.7mb
yellow open   filebeat-7.17.13-2023.12.27                             3Ff2VmfjT72ElvbbUC3VyA   1   1     267396            0     29.8mb         29.8mb
yellow open   filebeat-7.17.13-2024.01.02                             uuFIjZdkTYqO6RNEx9B9gg   1   1       3530            0      2.7mb          2.7mb
yellow open   filebeat-7.17.13-2024.01.01                             Kj2FCRBUQca5n3mg-yed7Q   1   1       2671            0      1.2mb          1.2mb
green  open   .async-search                                           owJdTDbeSgaNvlrmbMH4cw   1   0          0            0      3.4kb          3.4kb
yellow open   filebeat-7.17.13-2023.12.26                             MOlB9WyrSOaDDCf-CYJoJQ   1   1    2108355            0      235mb          235mb
2

I need some suggestion on the above because if I run it without the prefix it just works. I maybe incorrect on providing the value field "filebeat-*" because of which it is not working as expected.

Below is the example of the logs where it deleted the most of the things without the criteria.

2024-01-03 06:30:01,819 INFO      Preparing Action ID: 1, "delete_indices"
2024-01-03 06:30:01,819 INFO      Creating client object and testing connection
2024-01-03 06:30:01,820 INFO      Instantiating client object
2024-01-03 06:30:01,820 INFO      Testing client connectivity
2024-01-03 06:30:01,826 INFO      Successfully created Elasticsearch client object with provided settings
2024-01-03 06:30:01,828 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 7 days
2024-01-03 06:30:02,294 INFO      Deleting 11 selected indices: ['.kibana_7.17.13_001', 'logstash-2023.12.12-000001', '.tasks', '.apm-custom-link', 'logstash-2023.12.12-000001_202312', 'logstash-202312_202312', '.async-search', '.kibana_task_manager_7.17.13_001', '%{[@metadata][beat]}-%{[@metadata][version]}-2023.12.26', '%{[@metadata][beat]}-%{[@metadata][version]}-2023.12.25', '.apm-agent-configuration']
2024-01-03 06:30:02,295 INFO      ---deleting index .kibana_7.17.13_001
2024-01-03 06:30:02,295 INFO      ---deleting index logstash-2023.12.12-000001
2024-01-03 06:30:02,295 INFO      ---deleting index .tasks
2024-01-03 06:30:02,295 INFO      ---deleting index .apm-custom-link
2024-01-03 06:30:02,295 INFO      ---deleting index logstash-2023.12.12-000001_202312
2024-01-03 06:30:02,296 INFO      ---deleting index logstash-202312_202312
2024-01-03 06:30:02,296 INFO      ---deleting index .async-search
2024-01-03 06:30:02,296 INFO      ---deleting index .kibana_task_manager_7.17.13_001
2024-01-03 06:30:02,296 INFO      ---deleting index %{[@metadata][beat]}-%{[@metadata][version]}-2023.12.26
2024-01-03 06:30:02,296 INFO      ---deleting index %{[@metadata][beat]}-%{[@metadata][version]}-2023.12.25
2024-01-03 06:30:02,296 INFO      ---deleting index .apm-agent-configuration
2024-01-03 06:30:02,621 INFO      Action ID: 1, "delete_indices" completed.
2024-01-03 06:30:02,621 INFO      Job completed.
3

Have you tried to use just filebeat- instead of filebeat-*?

This is how it is show in the documentation.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.