Issues when running curator 8.0.17

Elastic Search
15

I really do not know why the value: "^\\.*" pattern isn't working for you:

actions:
  1:
    description: "This action should not affect anything. It's here to show filter behavior"
    action: open
    options:
      allow_ilm_indices: True
    filters:
      - filtertype: pattern
        kind: regex
        exclude: true
        value: "^\\.*"

Sample of debug log output from my own cluster running 8.17.0:

2025-02-05 11:04:42,769 DEBUG          curator.indexlist        iterate_filters:1452 Iterating over a list of filters
2025-02-05 11:04:42,769 DEBUG          curator.indexlist        iterate_filters:1457 All filters: [{'filtertype': 'pattern', 'kind': 'regex', 'exclude': True, 'value': '^\\.*'}]
2025-02-05 11:04:42,773 DEBUG          curator.indexlist        filter_by_regex:620  Filtering indices by regex
2025-02-05 11:04:42,773 DEBUG          curator.indexlist       empty_list_check:448  Checking for empty list
2025-02-05 11:04:42,773 DEBUG          curator.indexlist           working_list:459  Generating working list of indices
2025-02-05 11:04:42,773 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: .ds-metrics-kubernetes.node-default-2024.11.24-000002
2025-02-05 11:04:42,773 DEBUG          curator.indexlist       __not_actionable:58   Index .ds-metrics-kubernetes.node-default-2024.11.24-000002 is not actionable, removing from list.
2025-02-05 11:04:42,773 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: .ds-metrics-kubernetes.state_daemonset-k8s-2025.02.04-000004
2025-02-05 11:04:42,773 DEBUG          curator.indexlist       __not_actionable:58   Index .ds-metrics-kubernetes.state_daemonset-k8s-2025.02.04-000004 is not actionable, removing from list.
...

There's a lot more of the same that follows. As configured, this is filtering indices that start with a ..

Let's go back to my earlier recommendation to use a search_pattern. Since we're having some issues with the regex pattern not filtering indices that start with a . we will take the other recommended route.

Initially, you shared a warning level log message showing system indices that should not be accessed directly:

/usr/local/lib/python3.8/dist-packages/curator/indexlist.py:149: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .async-search, .kibana_7.12.0_001, .kibana_7.17.16_001, .reporting-2023-02-12, .reporting-2024-01-07, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default
  return self.client.indices.get_settings(index=to_csv(data))

This is only a warning, so it would not halt execution. But we should be in the habit of heeding these warnings and not directly accessing the named indices.

Additionally, the .geoip_databases index and any other system index should also be omitted from consideration, including .fleet*, .kibana*, .apm-agent-configuration, .async-search, .reporting-*, .triggered_watches, and .watches.

As such, I have taken the liberty of creating a search_pattern that includes all indices except the names and patterns preceded by a - (negating the pattern/name):

search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*,-.apm-agent-configuration,-.async-search,-.reporting-*,-.triggered_watches,-.watches'

Please make your actions.yml file look like this:

---
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
      search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*,-.apm-agent-configuration,-.async-search,-.reporting-*,-.triggered_watches,-.watches'
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

I have tested this with the --dry-run flag in my own cluster and it does match indices by age with this pattern as expected.

Using search_pattern effectively

Running a single action is simple and easy, and if you don't have too many indices it runs rather quickly. But keep in mind that with an age filter, Curator grabs metadata for all indices found by search_pattern before proceeding to the filters. On my own cluster, 529 indices were found by this search_pattern, and it took over 19 seconds to run, generating 2065 log lines in DEBUG mode. Curator did the work to gather the metadata on all 529 indices before filtering by age.

But if I only needed to operate on a subset of all indices? Or what if I wanted my logs to be much shorter?

By updating search_pattern to only gather indices starting with .ds-metrics-kubernetes*, I can simplify the search_pattern, shorten execution time, and reduce the log output size:

---
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
      search_pattern: '.ds-metrics-kubernetes*'
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

In my cluster, this shortened the list of indices needing metadata to 60, and execution time dropped to 2.6 seconds, generating only 284 log lines in DEBUG mode.

You could just as easily expand the search_pattern to include more indices you know you will be working on:

    search_pattern: '.ds-metrics-kubernetes*,.ds-metrics-system*'

The index count climbed to 82, execution time was 3.6s, and log line count was 371.

Please use this feature to your advantage!