Issues when running curator 8.0.17

I have recently upgraded curator v8.0.17 from curator v.5.8.3.

curator is not running.

PFB:
root@logsdev:~/curator# ./run.sh
Traceback (most recent call last):
File "/usr/local/bin/curator", line 8, in
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 299, in cli
run(ctx)
File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 223, in run
if ilm_action_skip(client, action_def):
UnboundLocalError: local variable 'client' referenced before assignment

I am getting the below error message in the curator log file:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/elastic_transport/_transport.py", line 342, in perform_request
resp = node.perform_request(
File "/usr/local/lib/python3.8/dist-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
raise err from e
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7faef953c0d0>: Failed to establish a new connection: [Errno 111] Connection refused)
2025-01-09 11:59:02,606 INFO elastic_transport.transport perform_request:372 GET http://127.0.0.1:9200/ [status:N/A duration:0.000s]
2025-01-09 11:59:02,606 WARNING elastic_transport.node_pool mark_dead:249 Node <Urllib3HttpNode(http://127.0.0.1:9200)> has failed for 4 times in a row, putting on 8 second timeout
2025-01-09 11:59:02,606 CRITICAL es_client.helpers.config get_client:318 Unable to establish client connection to Elasticsearch!
2025-01-09 11:59:02,606 CRITICAL es_client.helpers.config get_client:319 Exception encountered: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7faef958bdf0>: Failed to establish a new connection: [Errno 111] Connection refused))
2025-01-09 11:59:02,606 DEBUG curator.cli run:220 Fatal exception encountered:

Could you please check and advise on this ?

Thanks,
Padma.

Hi! Sorry to hear you're having some issues with Curator 8.x.

I can see that in the logs you've provided, Curator was unable to connect to http://127.0.0.1:9200, but I'm not certain that is the actual host name you wanted as this is the default value. The reason may be a change in the configuration file layout between Curator 5.x and 8.x:

Curator 5.x client configuration:

---
client:
  hosts:
    - 127.0.0.1

Curator 8.x client configuration:

---
elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9200

Please note that client is now a sub-key of elasticsearch.

If this change is not the reason you are unable to connect, please help me help you by providing a redacted version of your client configuration file.

When pasting pure text, code, or log lines, for maximum readability, please paste the content between triple back-tics on different lines, e.g.

```
type or paste code here
```

You can also use the pre-formatted text icon in the editor (</>) which will create those lines for you. Then paste to replace the type or paste code here bits.

It helps us to read what you've shared more quickly and accurately. Thank you!

1 Like

Thanks for the info .I will try to modify the config to support curator v8 .

I have tried with new config as below ,

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9301
    cloud_id:
    ca_certs:
    client_cert:
    client_key:
    verify_certs:
    request_timeout: 30
  other_settings:
    master_only: False
    username:
    password:
    api_key:
      id:
      api_key:
      token:

logging:
  loglevel: DEBUG
  logfile: /root/curator/logs/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

I got the below error in the log file:

2025-01-09 17:02:17,362 INFO      elastic_transport.transport        perform_request:349  HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.08 [status:404 duration:0.003s]
2025-01-09 17:02:17,366 INFO      elastic_transport.transport        perform_request:349  HEAD http://127.0.0.1:9301/_alias/heartbeat-7.12.0-2025.01.03-001292 [status:404 duration:0.003s]
2025-01-09 17:02:17,369 INFO      elastic_transport.transport        perform_request:349  HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.07 [status:404 duration:0.003s]
2025-01-09 17:02:17,372 INFO      elastic_transport.transport        perform_request:349  HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.06 [status:404 duration:0.003s]
2025-01-09 17:02:17,390 INFO      elastic_transport.transport        perform_request:349  GET http://127.0.0.1:9301/.geoip_databases,.kibana_task_manager_7.17.16_001,.monitoring-alerts-7,.monitoring-es-7-2025.01.03,.monitoring-es-7-2025.01.04,.monitoring-es-7-2025.01.05,.monitoring-es-7-2025.01.06,.monitoring-es-7-2025.01.07,.monitoring-es-7-2025.01.08,.monitoring-es-7-2025.01.09,.monitoring-kibana-7-2025.01.08,.monitoring-kibana-7-2025.01.09,.reporting-2022-04-03,.reporting-2024-03-17,.tasks,access-%5B2024.12.09%5D,access-%5B2024.12.30%5D,access-%5B2024.12.31%5D,access-%5B2025.01.09%5D,api_access-%5B2024.12.28%5D,api_access-%5B2024.12.29%5D,api_access-%5B2024.12.30%5D,api_access-%5B2025.01.01%5D,api_access-%5B2025.01.02%5D,api_access-%5B2025.01.03%5D,api_access-%5B2025.01.04%5D,heartbeat-7.12.0-2024.12.31-001289,heartbeat-7.12.0-2025.01.03-001292,heartbeat-7.12.0-2025.01.05-001294,heartbeat-7.12.0-2025.01.06-001295,heartbeat-7.12.0-2025.01.08-001297,ilm-history-2-000010,ohs-%5B2024.12.20%5D,ohs-%5B2024.12.21%5D,ohs-%5B2024.12.22%5D,ohs-%5B2024.12.23%5D,ohs-%5B2024.12.24%5D,ohs-%5B2024.12.25%5D,ohs-%5B2024.12.26%5D,ohs-%5B2024.12.31%5D,ohs-%5B2025.01.05%5D,ohs-%5B2025.01.06%5D,ohs-%5B2025.01.07%5D,ohs-%5B2025.01.08%5D,ohs-%5B2025.01.09%5D,oracle-2021.10.20,oracle-2021.10.21,oracle-2021.10.22,oracle-2021.10.23,oracle-2021.10.24,oracle-2021.10.25,oracle-2021.10.26,oracle-2021.10.27,oracle-2021.10.28,oracle-2021.10.29,oracle-2022.01.20,oracle-2022.01.21,oracle-2022.01.22,oracle-2022.01.23,oracle-2022.01.24,oracle-2022.01.25,oracle-2022.01.26,oracle-2022.01.27,oracle-2022.01.28,oracle-2022.01.29,oracle-2022.01.30,oracle-2022.01.31,oracle-2022.02.01,oracle-2022.02.02,oracle-2022.02.03,oracle-2022.02.04,oracle-2022.02.05,oracle-2022.02.06,oracle-2022.02.07,oracle-2022.02.08,oracle-2022.02.09,oracle-2022.02.10,oracle-2022.02.11,oracle-2022.02.12,oracle-2022.02.13,oracle-2022.02.14,oracle-2022.02.15,oracle-2022.02.16,oracle-2022.02.17,oracle-2022.02.18,oracle-2022.02.19,oracle-2022.02.27,oracle-2022.02.28,oracle-2022.03.01,oracle-2022.03.02,oracle-2022.03.03,oracle-2022.03.04,oracle-2022.03.05,oracle-2022.03.06,oracle-2022.03.07,oracle-2022.03.08,oracle-2022.03.09,oracle-2022.03.10,oracle-2022.03.11,oracle-2022.03.12,oracle-2022.03.13,oracle-2022.03.14,oracle-2022.03.15,oracle-2022.03.16,oracle-2022.03.17,oracle-2022.03.18,oracle-2022.03.19,oracle-2022.03.20,oracle-2022.03.21,oracle-2022.03.22,oracle-2022.03.23,oracle-2022.03.24,oracle-2022.03.25,oracle-2022.03.26,oracle-2022.03.27,oracle-2022.03.30,oracle-2022.03.31,oracle-2022.04.01,oracle-2022.04.02,oracle-2022.04.03,oracle-2022.04.04,oracle-2022.04.05,oracle-2022.04.06,oracle-2022.04.07,oracle-2022.04.08,ords-%5B2024.12.23%5D,ords-%5B2024.12.24%5D,ords-%5B2024.12.25%5D,ords-%5B2024.12.26%5D,ords-%5B2024.12.30%5D,ords-%5B2024.12.31%5D,ords-%5B2025.01.05%5D,ords-%5B2025.01.06%5D,ords-%5B2025.01.07%5D,ords-%5B2025.01.08%5D,st-active-users-%5B2024.12.24%5D,st-active-users-%5B2024.12.30%5D,st-active-users-%5B2024.12.31%5D,st-active-users-%5B2025.01.04%5D,st-active-users-%5B2025.01.05%5D,st-active-users-%5B2025.01.06%5D,st-active-users-%5B2025.01.07%5D,st-active-users-%5B2025.01.08%5D,st-active-users-%5B2025.01.09%5D/_settings [status:400 duration:0.017s]
2025-01-09 17:02:17,390 ERROR                curator.cli      exception_handler:91   Failed to complete action: delete_indices.  <class 'elasticsearch8.BadRequestError'>: BadRequestError(400, 'illegal_argument_exception', 'Indices [.geoip_databases] use and access is reserved for system operations')

FYI - Please find below for my existing config

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - node1.xxxx.com
    - node02.xxxx.com
    - node03.xxxx.com
  port: 9301
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile: /root/curator/logs/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Could you please check and let me know?

Thanks for your support.

Thanks.

While running the curator manually i am getting the below warning.

root@logsdev:~/curator# ./run.sh

/usr/local/lib/python3.8/dist-packages/curator/indexlist.py:149: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .async-search, .kibana_7.12.0_001, .kibana_7.17.16_001, .reporting-2023-02-12, .reporting-2024-01-07, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default
  return self.client.indices.get_settings(index=to_csv(data))

root@logsdev:~/curator# cat run.sh

#!/usr/bin/env bash

curator --config /root/curator/curator.config.yml /root/curator/actions.yml

Please find below my actions.yml file.

root@logsdev:~/curator# cat actions.yml

--
#
# We don't want the disk to fill up too much so use the following action set to clear down
# unnecessary data.
#
# You'll need to change this or add to it if you ever do rollover indexes instead.
#
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

root@logsdev:~/curator#

could you please check ?

Thanks,
Padma.

Indices [.geoip_databases] use and access is reserved for system operations

First, you appear to have only a single age filter. Granted, this will look for a ####.##.## pattern in indices and try to match that to Year.Month.Day numbers for age calculation, but it is searching all indices first. This is causing Curator to evaluate system indices, i.e. those that are preceded by dots/periods.

Since you also appear to be using a newer version of Elasticsearch that is halting operations against protected system indices, you must exclude them from consideration to avoid this 400 error.

There are two ways to exclude system indices. You can use either or both, but they must be used for each defined action.

Exclude all dot prefixed indices
Make your first filter a regex like this:

    - filtertype: pattern
      kind: regex
      exclude: true
      value: "^\\.*"

You can add as many filters as you want, but they will always be processed in the order in which they appear in the YAML file.

Use a search_pattern (now called multi-target syntax)
You can use an Elasticsearch multi-target syntax to include/exclude indices this way:

---
actions:
  1:
    action: delete_indices
    description: "Example of search_pattern"
    options:
      search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*'

This search pattern begins by including all (*) indices, but then explicitly excludes (-) .geoip_databases, anything starting with .fleet, and anything starting with .kibana. You must start with some indices before you can exclude. In this example, the wildcard for all is used first, and then the exclusions are listed.

1 Like

It's working now .but i am seeing the below on the log file.

on': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-14 10:56:17,817 DEBUG          curator.indexlist        iterate_filters:1466 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-14 10:56:17,817 DEBUG          curator.indexlist        iterate_filters:1472 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}
2025-01-14 10:56:17,818 DEBUG          curator.indexlist        iterate_filters:1473 Pre-instance: []
2025-01-14 10:56:17,818 DEBUG          curator.indexlist          filter_by_age:683  Filtering indices by age
2025-01-14 10:56:17,818 DEBUG          curator.indexlist     get_index_settings:328  Getting index settings -- BEGIN
2025-01-14 10:56:17,818 DEBUG          curator.indexlist       empty_list_check:448  Checking for empty list
2025-01-14 10:56:17,818 INFO                 curator.cli      exception_handler:78   Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2025-01-14 10:56:17,818 INFO                 curator.cli                    run:236  Action ID: 1, "delete_indices" completed.
2025-01-14 10:56:17,818 INFO                 curator.cli                    run:237  All actions completed.

Please find the below my config files:

root@logsdev:~/curator# cat actions.yml
---
#
# We don't want the disk to fill up too much so use the following action set to clear down
# unnecessary data.
#
# You'll need to change this or add to it if you ever do rollover indexes instead.
#
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: '.'  # Exclude indices starting with a dot (system indices)
      exclude: true
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

curator.config.yml

root@logsdev:~/curator# cat curator.config.yml
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9301
      - http://logsnode02.corehr.com:9301
      - http://logsnode03.corehr.com:9301
    cloud_id:
    ca_certs:
    client_cert:
    client_key:
    verify_certs:
    request_timeout: 30
  other_settings:
    master_only: False
    username:
    password:
    api_key:
      id:
      api_key:
      token:

logging:
  loglevel: DEBUG
  logfile: /root/curator/logs/curator.log
  logformat: default
  blacklist: ['elastic_transport', 'urllib3']

I dont have the logs older than 31 days in the indices .Due to that i am getting the empty list.

Please confirm.

Thanks.

This is not what I said to use. The prefix pattern is still a regular expression. It just adds the ^ anchor before whatever you put in value. A plain . will match any character. This is why all of your indices are being matched and excluded, resulting the empty list error.

What I shared and expect you to use is:

    - filtertype: pattern
      kind: regex
      exclude: true
      value: "^\\.*"

This is a pure regular expression that includes the ^ anchor to match the beginning of the line. It then escapes the period \\ which says to match an actual period and not just any character.

Replace the incorrect pattern filter with the one I already provided and everything will work.

Thank you .
I have updated my config file as below .but still i am getting the same log file.

please find below my config:

root@logsdev:~/curator# cat actions.yml
---
#
# We don't want the disk to fill up too much so use the following action set to clear down
# unnecessary data.
#
# You'll need to change this or add to it if you ever do rollover indexes instead.
#
actions:
  1:
    action: delete_indices
    description: "Delete indices older than 31 days"
    options:
      allow_ilm_indices: true
      ignore_empty_list: true
      disable_action: false
    filters:
    - filtertype: pattern
      kind: regex
      exclude: true
      value: "^\\.*"
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 31

Output log file:

2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index st-active-users-[2025.01.04] is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: oracle-2022.03.09
2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index oracle-2022.03.09 is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: oracle-2022.03.08
2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index oracle-2022.03.08 is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: oracle-2022.03.07
2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index oracle-2022.03.07 is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        filter_by_regex:637  Filter by regex: Index: oracle-2022.03.06
2025-01-14 14:23:33,799 DEBUG          curator.indexlist       __not_actionable:58   Index oracle-2022.03.06 is not actionable, removing from list.
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        iterate_filters:1475 Post-instance: []
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        iterate_filters:1459 Top of the loop: []
2025-01-14 14:23:33,799 DEBUG          curator.indexlist        iterate_filters:1460 Un-parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2025-01-14 14:23:33,800 DEBUG     es_client.helpers.schemacheck               __init__:60   Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'size_behavior': Any(<class 'str'>, msg=None), 'size_threshold': Any(Coerce(float, msg=None), msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7fcf34856430>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']")}
2025-01-14 14:23:33,800 DEBUG     es_client.helpers.schemacheck               __init__:62   "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2025-01-14 14:23:33,800 DEBUG          curator.indexlist        iterate_filters:1466 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2025-01-14 14:23:33,800 DEBUG          curator.indexlist        iterate_filters:1472 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2025-01-14 14:23:33,800 DEBUG          curator.indexlist        iterate_filters:1473 Pre-instance: []
2025-01-14 14:23:33,801 DEBUG          curator.indexlist          filter_by_age:683  Filtering indices by age
2025-01-14 14:23:33,801 DEBUG          curator.indexlist     get_index_settings:328  Getting index settings -- BEGIN
2025-01-14 14:23:33,801 DEBUG          curator.indexlist       empty_list_check:448  Checking for empty list
2025-01-14 14:23:33,801 INFO                 curator.cli      exception_handler:78   Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2025-01-14 14:23:33,801 INFO                 curator.cli                    run:236  Action ID: 1, "delete_indices" completed.
2025-01-14 14:23:33,801 INFO                 curator.cli                    run:237  All actions completed.

Could you please check on this?

Regards,
Padma.