Issues when running curator 8.0.17

I have recently upgraded curator v8.0.17 from curator v.5.8.3.

curator is not running.

PFB:
root@logsdev:~/curator# ./run.sh
Traceback (most recent call last):
File "/usr/local/bin/curator", line 8, in
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 299, in cli
run(ctx)
File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 223, in run
if ilm_action_skip(client, action_def):
UnboundLocalError: local variable 'client' referenced before assignment

I am getting the below error message in the curator log file:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/elastic_transport/_transport.py", line 342, in perform_request
resp = node.perform_request(
File "/usr/local/lib/python3.8/dist-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
raise err from e
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7faef953c0d0>: Failed to establish a new connection: [Errno 111] Connection refused)
2025-01-09 11:59:02,606 INFO elastic_transport.transport perform_request:372 GET http://127.0.0.1:9200/ [status:N/A duration:0.000s]
2025-01-09 11:59:02,606 WARNING elastic_transport.node_pool mark_dead:249 Node <Urllib3HttpNode(http://127.0.0.1:9200)> has failed for 4 times in a row, putting on 8 second timeout
2025-01-09 11:59:02,606 CRITICAL es_client.helpers.config get_client:318 Unable to establish client connection to Elasticsearch!
2025-01-09 11:59:02,606 CRITICAL es_client.helpers.config get_client:319 Exception encountered: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7faef958bdf0>: Failed to establish a new connection: [Errno 111] Connection refused))
2025-01-09 11:59:02,606 DEBUG curator.cli run:220 Fatal exception encountered:

Could you please check and advise on this ?

Thanks,
Padma.

Hi! Sorry to hear you're having some issues with Curator 8.x.

I can see that in the logs you've provided, Curator was unable to connect to http://127.0.0.1:9200, but I'm not certain that is the actual host name you wanted as this is the default value. The reason may be a change in the configuration file layout between Curator 5.x and 8.x:

Curator 5.x client configuration:

---
client:
  hosts:
    - 127.0.0.1

Curator 8.x client configuration:

---
elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9200

Please note that client is now a sub-key of elasticsearch.

If this change is not the reason you are unable to connect, please help me help you by providing a redacted version of your client configuration file.

When pasting pure text, code, or log lines, for maximum readability, please paste the content between triple back-tics on different lines, e.g.

```
type or paste code here
```

You can also use the pre-formatted text icon in the editor (</>) which will create those lines for you. Then paste to replace the type or paste code here bits.

It helps us to read what you've shared more quickly and accurately. Thank you!

1 Like

Thanks for the info .I will try to modify the config to support curator v8 .

I have tried with new config as below ,

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9301
    cloud_id:
    ca_certs:
    client_cert:
    client_key:
    verify_certs:
    request_timeout: 30
  other_settings:
    master_only: False
    username:
    password:
    api_key:
      id:
      api_key:
      token:

logging:
  loglevel: DEBUG
  logfile: /root/curator/logs/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

I got the below error in the log file:

2025-01-09 17:02:17,362 INFO      elastic_transport.transport        perform_request:349  HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.08 [status:404 duration:0.003s]
2025-01-09 17:02:17,366 INFO      elastic_transport.transport        perform_request:349  HEAD http://127.0.0.1:9301/_alias/heartbeat-7.12.0-2025.01.03-001292 [status:404 duration:0.003s]
2025-01-09 17:02:17,369 INFO      elastic_transport.transport        perform_request:349  HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.07 [status:404 duration:0.003s]
2025-01-09 17:02:17,372 INFO      elastic_transport.transport        perform_request:349  HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.06 [status:404 duration:0.003s]
2025-01-09 17:02:17,390 INFO      elastic_transport.transport        perform_request:349  GET http://127.0.0.1:9301/.geoip_databases,.kibana_task_manager_7.17.16_001,.monitoring-alerts-7,.monitoring-es-7-2025.01.03,.monitoring-es-7-2025.01.04,.monitoring-es-7-2025.01.05,.monitoring-es-7-2025.01.06,.monitoring-es-7-2025.01.07,.monitoring-es-7-2025.01.08,.monitoring-es-7-2025.01.09,.monitoring-kibana-7-2025.01.08,.monitoring-kibana-7-2025.01.09,.reporting-2022-04-03,.reporting-2024-03-17,.tasks,access-%5B2024.12.09%5D,access-%5B2024.12.30%5D,access-%5B2024.12.31%5D,access-%5B2025.01.09%5D,api_access-%5B2024.12.28%5D,api_access-%5B2024.12.29%5D,api_access-%5B2024.12.30%5D,api_access-%5B2025.01.01%5D,api_access-%5B2025.01.02%5D,api_access-%5B2025.01.03%5D,api_access-%5B2025.01.04%5D,heartbeat-7.12.0-2024.12.31-001289,heartbeat-7.12.0-2025.01.03-001292,heartbeat-7.12.0-2025.01.05-001294,heartbeat-7.12.0-2025.01.06-001295,heartbeat-7.12.0-2025.01.08-001297,ilm-history-2-000010,ohs-%5B2024.12.20%5D,ohs-%5B2024.12.21%5D,ohs-%5B2024.12.22%5D,ohs-%5B2024.12.23%5D,ohs-%5B2024.12.24%5D,ohs-%5B2024.12.25%5D,ohs-%5B2024.12.26%5D,ohs-%5B2024.12.31%5D,ohs-%5B2025.01.05%5D,ohs-%5B2025.01.06%5D,ohs-%5B2025.01.07%5D,ohs-%5B2025.01.08%5D,ohs-%5B2025.01.09%5D,oracle-2021.10.20,oracle-2021.10.21,oracle-2021.10.22,oracle-2021.10.23,oracle-2021.10.24,oracle-2021.10.25,oracle-2021.10.26,oracle-2021.10.27,oracle-2021.10.28,oracle-2021.10.29,oracle-2022.01.20,oracle-2022.01.21,oracle-2022.01.22,oracle-2022.01.23,oracle-2022.01.24,oracle-2022.01.25,oracle-2022.01.26,oracle-2022.01.27,oracle-2022.01.28,oracle-2022.01.29,oracle-2022.01.30,oracle-2022.01.31,oracle-2022.02.01,oracle-2022.02.02,oracle-2022.02.03,oracle-2022.02.04,oracle-2022.02.05,oracle-2022.02.06,oracle-2022.02.07,oracle-2022.02.08,oracle-2022.02.09,oracle-2022.02.10,oracle-2022.02.11,oracle-2022.02.12,oracle-2022.02.13,oracle-2022.02.14,oracle-2022.02.15,oracle-2022.02.16,oracle-2022.02.17,oracle-2022.02.18,oracle-2022.02.19,oracle-2022.02.27,oracle-2022.02.28,oracle-2022.03.01,oracle-2022.03.02,oracle-2022.03.03,oracle-2022.03.04,oracle-2022.03.05,oracle-2022.03.06,oracle-2022.03.07,oracle-2022.03.08,oracle-2022.03.09,oracle-2022.03.10,oracle-2022.03.11,oracle-2022.03.12,oracle-2022.03.13,oracle-2022.03.14,oracle-2022.03.15,oracle-2022.03.16,oracle-2022.03.17,oracle-2022.03.18,oracle-2022.03.19,oracle-2022.03.20,oracle-2022.03.21,oracle-2022.03.22,oracle-2022.03.23,oracle-2022.03.24,oracle-2022.03.25,oracle-2022.03.26,oracle-2022.03.27,oracle-2022.03.30,oracle-2022.03.31,oracle-2022.04.01,oracle-2022.04.02,oracle-2022.04.03,oracle-2022.04.04,oracle-2022.04.05,oracle-2022.04.06,oracle-2022.04.07,oracle-2022.04.08,ords-%5B2024.12.23%5D,ords-%5B2024.12.24%5D,ords-%5B2024.12.25%5D,ords-%5B2024.12.26%5D,ords-%5B2024.12.30%5D,ords-%5B2024.12.31%5D,ords-%5B2025.01.05%5D,ords-%5B2025.01.06%5D,ords-%5B2025.01.07%5D,ords-%5B2025.01.08%5D,st-active-users-%5B2024.12.24%5D,st-active-users-%5B2024.12.30%5D,st-active-users-%5B2024.12.31%5D,st-active-users-%5B2025.01.04%5D,st-active-users-%5B2025.01.05%5D,st-active-users-%5B2025.01.06%5D,st-active-users-%5B2025.01.07%5D,st-active-users-%5B2025.01.08%5D,st-active-users-%5B2025.01.09%5D/_settings [status:400 duration:0.017s]
2025-01-09 17:02:17,390 ERROR                curator.cli      exception_handler:91   Failed to complete action: delete_indices.  <class 'elasticsearch8.BadRequestError'>: BadRequestError(400, 'illegal_argument_exception', 'Indices [.geoip_databases] use and access is reserved for system operations')

FYI - Please find below for my existing config

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - node1.xxxx.com
    - node02.xxxx.com
    - node03.xxxx.com
  port: 9301
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile: /root/curator/logs/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Could you please check and let me know?

Thanks for your support.

Thanks.

While running the curator manually i am getting the below warning.

root@logsdev:~/curator# ./run.sh

/usr/local/lib/python3.8/dist-packages/curator/indexlist.py:149: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .async-search, .kibana_7.12.0_001, .kibana_7.17.16_001, .reporting-2023-02-12, .reporting-2024-01-07, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default
  return self.client.indices.get_settings(index=to_csv(data))

root@logsdev:~/curator# cat run.sh

#!/usr/bin/env bash

curator --config /root/curator/curator.config.yml /root/curator/actions.yml

Please find below my actions.yml file.

root@logsdev:~/curator# cat actions.yml

--
#
# We don't want the disk to fill up too much so use the following action set to clear down
# unnecessary data.
#
# You'll need to change this or add to it if you ever do rollover indexes instead.
#
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

root@logsdev:~/curator#

could you please check ?

Thanks,
Padma.

Indices [.geoip_databases] use and access is reserved for system operations

First, you appear to have only a single age filter. Granted, this will look for a ####.##.## pattern in indices and try to match that to Year.Month.Day numbers for age calculation, but it is searching all indices first. This is causing Curator to evaluate system indices, i.e. those that are preceded by dots/periods.

Since you also appear to be using a newer version of Elasticsearch that is halting operations against protected system indices, you must exclude them from consideration to avoid this 400 error.

There are two ways to exclude system indices. You can use either or both, but they must be used for each defined action.

Exclude all dot prefixed indices
Make your first filter a regex like this:

    - filtertype: pattern
      kind: regex
      exclude: true
      value: "^\\.*"

You can add as many filters as you want, but they will always be processed in the order in which they appear in the YAML file.

Use a search_pattern (now called multi-target syntax)
You can use an Elasticsearch multi-target syntax to include/exclude indices this way:

---
actions:
  1:
    action: delete_indices
    description: "Example of search_pattern"
    options:
      search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*'

This search pattern begins by including all (*) indices, but then explicitly excludes (-) .geoip_databases, anything starting with .fleet, and anything starting with .kibana. You must start with some indices before you can exclude. In this example, the wildcard for all is used first, and then the exclusions are listed.

1 Like

It's working now .but i am seeing the below on the log file.

on': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-14 10:56:17,817 DEBUG          curator.indexlist        iterate_filters:1466 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-14 10:56:17,817 DEBUG          curator.indexlist        iterate_filters:1472 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-14 10:56:17,818 DEBUG          curator.indexlist        iterate_filters:1473 Pre-instance: []
2025-01-14 10:56:17,818 DEBUG          curator.indexlist          filter_by_age:683  Filtering indices by age
2025-01-14 10:56:17,818 DEBUG          curator.indexlist     get_index_settings:328  Getting index settings -- BEGIN
2025-01-14 10:56:17,818 DEBUG          curator.indexlist       empty_list_check:448  Checking for empty list
2025-01-14 10:56:17,818 INFO                 curator.cli      exception_handler:78   Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2025-01-14 10:56:17,818 INFO                 curator.cli                    run:236  Action ID: 1, "delete_indices" completed.
2025-01-14 10:56:17,818 INFO                 curator.cli                    run:237  All actions completed.

Please find the below my config files:

root@logsdev:~/curator# cat actions.yml
---
#
# We don't want the disk to fill up too much so use the following action set to clear down
# unnecessary data.
#
# You'll need to change this or add to it if you ever do rollover indexes instead.
#
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: '.'  # Exclude indices starting with a dot (system indices)
      exclude: true
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

curator.config.yml

root@logsdev:~/curator# cat curator.config.yml
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9301
      - http://logsnode02.corehr.com:9301
      - http://logsnode03.corehr.com:9301
    cloud_id:
    ca_certs:
    client_cert:
    client_key:
    verify_certs:
    request_timeout: 30
  other_settings:
    master_only: False
    username:
    password:
    api_key:
      id:
      api_key:
      token:

logging:
  loglevel: DEBUG
  logfile: /root/curator/logs/curator.log
  logformat: default
  blacklist: ['elastic_transport', 'urllib3']

I dont have the logs older than 31 days in the indices .Due to that i am getting the empty list.

Please confirm.

Thanks.

This is not what I said to use. The prefix pattern is still a regular expression. It just adds the ^ anchor before whatever you put in value. A plain . will match any character. This is why all of your indices are being matched and excluded, resulting the empty list error.

What I shared and expect you to use is:

    - filtertype: pattern
      kind: regex
      exclude: true
      value: "^\\.*"

This is a pure regular expression that includes the ^ anchor to match the beginning of the line. It then escapes the period \\ which says to match an actual period and not just any character.

Replace the incorrect pattern filter with the one I already provided and everything will work.

Thank you .
I have updated my config file as below .but still i am getting the same log file.

please find below my config:

root@logsdev:~/curator# cat actions.yml
---
#
# We don't want the disk to fill up too much so use the following action set to clear down
# unnecessary data.
#
# You'll need to change this or add to it if you ever do rollover indexes instead.
#
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
    filters:
    - filtertype: pattern
      kind: regex
      exclude: true
      value: "^\\.*"
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

Output log file:

2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index st-active-users-[2025.01.04] is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: oracle-2022.03.09
2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index oracle-2022.03.09 is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: oracle-2022.03.08
2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index oracle-2022.03.08 is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: oracle-2022.03.07
2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index oracle-2022.03.07 is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: oracle-2022.03.06
2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index oracle-2022.03.06 is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        iterate_filters:1475 Post-instance: []
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        iterate_filters:1459 Top of the loop: []
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        iterate_filters:1460 Un-parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2025-01-14 14:23:33,800 DEBUG     es_client.helpers.schemacheck               __init__:60   Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'size_behavior': Any(<class 'str'>, msg=None), 'size_threshold': Any(Coerce(float, msg=None), msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7fcf34856430>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']")}
2025-01-14 14:23:33,800 DEBUG     es_client.helpers.schemacheck               __init__:62   "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2025-01-14 14:23:33,800 DEBUG          curator.indexlist        iterate_filters:1466 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2025-01-14 14:23:33,800 DEBUG          curator.indexlist        iterate_filters:1472 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2025-01-14 14:23:33,800 DEBUG          curator.indexlist        iterate_filters:1473 Pre-instance: []
2025-01-14 14:23:33,801 DEBUG          curator.indexlist          filter_by_age:683  Filtering indices by age
2025-01-14 14:23:33,801 DEBUG          curator.indexlist     get_index_settings:328  Getting index settings -- BEGIN
2025-01-14 14:23:33,801 DEBUG          curator.indexlist       empty_list_check:448  Checking for empty list
2025-01-14 14:23:33,801 INFO                 curator.cli      exception_handler:78   Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2025-01-14 14:23:33,801 INFO                 curator.cli                    run:236  Action ID: 1, "delete_indices" completed.
2025-01-14 14:23:33,801 INFO                 curator.cli                    run:237  All actions completed.

Could you please check on this?

Regards,
Padma.

Sorry for the delay. I was out of office.

I cannot see enough of what happened here. Something seems amiss, however, as it is filtering all indices and not just ones starting with a .

I'm not sure why it's not working as expected. Please try changing:

value: "^\\.*"

to

value: "^\.*"

And add the --dry-run flag to your command-line so it shows what is being filtered, but does not delete anything yet.

No ,It didnt work.i am getting the same schema error:
Schema error: extra keys not allowed @ data['elasticsearch'].
It was not deleting the older indices .

n': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-30 17:45:05,545 DEBUG     es_client.helpers.schemacheck               __init__:60   Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'size_behavior': Any(<class 'str'>, msg=None), 'size_threshold': Any(Coerce(float, msg=None), msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7fe2f88d94c0>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']")}
2025-01-30 17:45:05,545 DEBUG     es_client.helpers.schemacheck               __init__:62   "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-30 17:45:05,545 DEBUG          curator.indexlist        iterate_filters:1466 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-30 17:45:05,546 DEBUG          curator.indexlist        iterate_filters:1472 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-30 17:45:05,546 DEBUG          curator.indexlist        iterate_filters:1473 Pre-instance: []
2025-01-30 17:45:05,546 DEBUG          curator.indexlist          filter_by_age:683  Filtering indices by age
2025-01-30 17:45:05,546 DEBUG          curator.indexlist     get_index_settings:328  Getting index settings -- BEGIN
2025-01-30 17:45:05,546 DEBUG          curator.indexlist       empty_list_check:448  Checking for empty list
2025-01-30 17:45:05,546 INFO                 curator.cli      exception_handler:78   Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2025-01-30 17:45:05,546 INFO                 curator.cli                    run:236  Action ID: 1, "delete_indices" completed.
2025-01-30 17:45:05,546 INFO                 curator.cli                    run:237  All actions completed.
Schema error: extra keys not allowed @ data['elasticsearch']
Schema error: extra keys not allowed @ data['elasticsearch']

Please find the attached config files

root@logsdev:~/curator# cat curator.config.yml
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9301
      - http://logsnode02.corehr.com:9301
      - http://logsnode03.corehr.com:9301
    cloud_id:
    ca_certs:
    client_cert:
    client_key:
    verify_certs:
    request_timeout: 30
  other_settings:
    master_only: False
    username:
    password:
    api_key:
      id:
      api_key:
      token:
logging:
  loglevel: DEBUG
  logfile: /root/curator/logs/curator.log
  logformat: default
  blacklist: ['elastic_transport', 'urllib3']

action file
root@logsdev:~/curator# cat actions.yml
---
#
# We don't want the disk to fill up too much so use the following action set to clear down
# unnecessary data.
#
# You'll need to change this or add to it if you ever do rollover indexes instead.
#
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
    filters:
    - filtertype: pattern
      kind: regex
      exclude: true
      value: "^\\.*"
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

could you please check on this ?
Thanks.

I think ,curator version 8.0.17 is not compatible with ES version 8.

Curator and Elasticsearch 8 - Elastic Stack / Elasticsearch - Discuss the Elastic Stack

Can we go with ILM policy ?

Please confirm .

Thanks.

The post you linked is dated October 2022. Curator 8.0.0 was released January 31, 2023.

I am facing issue with the age filter .its not removing the older indices and also im facing the schema error especially with the cron jobs .
I am using curator version 8.0.17 and ES version 8.0.1.

Thanks.

I really do not know why the value: "^\\.*" pattern isn't working for you:

actions:
  1:
    description: "This action should not affect anything. It's here to show filter behavior"
    action: open
    options:
      allow_ilm_indices: True
    filters:
      - filtertype: pattern
        kind: regex
        exclude: true
        value: "^\\.*"

Sample of debug log output from my own cluster running 8.17.0:

2025-02-05 11:04:42,769 DEBUG          curator.indexlist        iterate_filters:1452 Iterating over a list of filters
2025-02-05 11:04:42,769 DEBUG          curator.indexlist        iterate_filters:1457 All filters: [{'filtertype': 'pattern', 'kind': 'regex', 'exclude': True, 'value': '^\\.*'}]
2025-02-05 11:04:42,773 DEBUG          curator.indexlist        filter_by_regex:620  Filtering indices by regex
2025-02-05 11:04:42,773 DEBUG          curator.indexlist       empty_list_check:448  Checking for empty list
2025-02-05 11:04:42,773 DEBUG          curator.indexlist           working_list:459  Generating working list of indices
2025-02-05 11:04:42,773 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: .ds-metrics-kubernetes.node-default-2024.11.24-000002
2025-02-05 11:04:42,773 DEBUG          curator.indexlist       __not_actionable:58   Index .ds-metrics-kubernetes.node-default-2024.11.24-000002 is not actionable, removing from list.
2025-02-05 11:04:42,773 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: .ds-metrics-kubernetes.state_daemonset-k8s-2025.02.04-000004
2025-02-05 11:04:42,773 DEBUG          curator.indexlist       __not_actionable:58   Index .ds-metrics-kubernetes.state_daemonset-k8s-2025.02.04-000004 is not actionable, removing from list.
...

There's a lot more of the same that follows. As configured, this is filtering indices that start with a ..

Let's go back to my earlier recommendation to use a search_pattern. Since we're having some issues with the regex pattern not filtering indices that start with a . we will take the other recommended route.

Initially, you shared a warning level log message showing system indices that should not be accessed directly:

/usr/local/lib/python3.8/dist-packages/curator/indexlist.py:149: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .async-search, .kibana_7.12.0_001, .kibana_7.17.16_001, .reporting-2023-02-12, .reporting-2024-01-07, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default
  return self.client.indices.get_settings(index=to_csv(data))

This is only a warning, so it would not halt execution. But we should be in the habit of heeding these warnings and not directly accessing the named indices.

Additionally, the .geoip_databases index and any other system index should also be omitted from consideration, including .fleet*, .kibana*, .apm-agent-configuration, .async-search, .reporting-*, .triggered_watches, and .watches.

As such, I have taken the liberty of creating a search_pattern that includes all indices except the names and patterns preceded by a - (negating the pattern/name):

search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*,-.apm-agent-configuration,-.async-search,-.reporting-*,-.triggered_watches,-.watches'

Please make your actions.yml file look like this:

---
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
      search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*,-.apm-agent-configuration,-.async-search,-.reporting-*,-.triggered_watches,-.watches'
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

I have tested this with the --dry-run flag in my own cluster and it does match indices by age with this pattern as expected.

Using search_pattern effectively

Running a single action is simple and easy, and if you don't have too many indices it runs rather quickly. But keep in mind that with an age filter, Curator grabs metadata for all indices found by search_pattern before proceeding to the filters. On my own cluster, 529 indices were found by this search_pattern, and it took over 19 seconds to run, generating 2065 log lines in DEBUG mode. Curator did the work to gather the metadata on all 529 indices before filtering by age.

But if I only needed to operate on a subset of all indices? Or what if I wanted my logs to be much shorter?

By updating search_pattern to only gather indices starting with .ds-metrics-kubernetes*, I can simplify the search_pattern, shorten execution time, and reduce the log output size:

---
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
      search_pattern: '.ds-metrics-kubernetes*'
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

In my cluster, this shortened the list of indices needing metadata to 60, and execution time dropped to 2.6 seconds, generating only 284 log lines in DEBUG mode.

You could just as easily expand the search_pattern to include more indices you know you will be working on:

    search_pattern: '.ds-metrics-kubernetes*,.ds-metrics-system*'

The index count climbed to 82, execution time was 3.6s, and log line count was 371.

Please use this feature to your advantage!

Your schema error is caused by the last line:

action file should not be here. That is the extra key that should not be there.

Thanks for the info .
I have added the search pattern and i could able to deleted the older indices.

please find below my action file:

---
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
      search_pattern: '*,-.geoip_databases,-.apm-custom-link*,-.fleet*,-.tasks*,-.kibana*,-.monitoring-alerts-7*,-.reporting*,-.apm-agent-configuration,-.async-search,-.reporting-*,-.triggered_watches,-.watches'
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

It's working fine manually .
I have scheduled cronjob to run this task:
* * * * * /root/curator/run.sh >> /root/curator/logs/curator.log 2>&1

if i am rununing via cronjob im getting the same schema error .

2025-02-06 11:17:48,803 DEBUG          curator.indexlist            __excludify:77   Removed from actionable list: Index "st-active-users-[2025.01.28]" age (1738022400), direction: "older", point of reference, (1736162268)
2025-02-06 11:17:48,803 DEBUG          curator.indexlist        iterate_filters:1475 Post-instance: []
2025-02-06 11:17:48,803 DEBUG                curator.cli         process_action:168  Pre Instantiation Action kwargs: {'master_timeout': 30}
2025-02-06 11:17:48,803 DEBUG     curator.actions.delete_indices               __init__:32   master_timeout value: 30s
2025-02-06 11:17:48,803 DEBUG                curator.cli         process_action:174  Doing the action here.
2025-02-06 11:17:48,803 DEBUG          curator.indexlist       empty_list_check:448  Checking for empty list
2025-02-06 11:17:48,803 INFO                 curator.cli      exception_handler:78   Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2025-02-06 11:17:48,803 INFO                 curator.cli                    run:236  Action ID: 1, "delete_indices" completed.
2025-02-06 11:17:48,803 INFO                 curator.cli                    run:237  All actions completed.
Schema error: extra keys not allowed @ data['elasticsearch']
Schema error: extra keys not allowed @ data['elasticsearch']
Schema error: extra keys not allowed @ data['elasticsearch']
Schema error: extra keys not allowed @ data['elasticsearch']
Schema error: extra keys not allowed @ data['elasticsearch']
Schema error: extra keys not allowed @ data['elasticsearch']
Schema error: extra keys not allowed @ data['elasticsearch']
Schema error: extra keys not allowed @ data['elasticsearch']

Could you please advise on this ?

Thanks.

The issue is fixed now .Thank you .

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.