# Issues when running curator 8.0.17 **URL:** \ **Category:** Elastic Search\ **Tags:** curator\ **Created:** [January 9, 2025, 12:42pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996 "2025-01-09T12:42:39Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [January 9, 2025, 12:42pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/1 "2025-01-09T12:42:39Z")
I have recently upgraded curator v8.0.17 from curator v.5.8.3. **curator is not running.** PFB: root@logsdev:~/curator# ./run.sh Traceback (most recent call last): File "/usr/local/bin/curator", line 8, in sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in **call** return self.main(\*args, \*\*kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, \*\*ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke return \_\_callback(\*args, \*\*kwargs) File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 33, in new\_func return f(get\_current\_context(), \*args, \*\*kwargs) File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 299, in cli run(ctx) File "/usr/local/lib/python3.8/dist-packages/curator/cli.py", line 223, in run if ilm\_action\_skip(client, action\_def): UnboundLocalError: local variable 'client' referenced before assignment **I am getting the below error message in the curator log file:** Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/elastic\_transport/\_transport.py", line 342, in perform\_request resp = node.perform\_request( File "/usr/local/lib/python3.8/dist-packages/elastic\_transport/\_node/\_http\_urllib3.py", line 202, in perform\_request raise err from e elastic\_transport.ConnectionError: Connection error caused by: NewConnectionError(\: Failed to establish a new connection: [Errno 111] Connection refused) 2025-01-09 11:59:02,606 INFO elastic\_transport.transport perform\_request:372 GET [http://127.0.0.1:9200/](http://127.0.0.1:9200/) [status:N/A duration:0.000s] 2025-01-09 11:59:02,606 WARNING elastic\_transport.node\_pool mark\_dead:249 Node \ has failed for 4 times in a row, putting on 8 second timeout 2025-01-09 11:59:02,606 CRITICAL es\_client.helpers.config get\_client:318 Unable to establish client connection to Elasticsearch! 2025-01-09 11:59:02,606 CRITICAL es\_client.helpers.config get\_client:319 Exception encountered: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(\: Failed to establish a new connection: [Errno 111] Connection refused)) 2025-01-09 11:59:02,606 DEBUG curator.cli run:220 Fatal exception encountered: Could you please check and advise on this ? Thanks, Padma. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [January 9, 2025, 3:49pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/2 "2025-01-09T15:49:33Z")
Hi! Sorry to hear you're having some issues with Curator 8.x. I can see that in the logs you've provided, Curator was unable to connect to `http://127.0.0.1:9200`, but I'm not certain that is the actual host name you wanted as this is the default value. The reason may be a change in the configuration file layout between Curator 5.x and 8.x: Curator [5.x client configuration](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/configfile.html): ```yaml --- client: hosts: - 127.0.0.1 ``` Curator [8.x client configuration](https://www.elastic.co/guide/en/elasticsearch/client/curator/8.0/configfile.html): ```yaml --- elasticsearch: client: hosts: - http://127.0.0.1:9200 ``` Please note that `client` is now a sub-key of `elasticsearch`. If this change is not the reason you are unable to connect, please help me help you by providing a redacted version of your client configuration file. When pasting pure text, code, or log lines, for maximum readability, please paste the content between triple back-tics on different lines, e.g. ```` ``` type or paste code here ``` ```` You can also use the pre-formatted text icon in the editor (``) which will create those lines for you. Then paste to replace the `type or paste code here` bits. It helps us to read what you've shared more quickly and accurately. Thank you! ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [January 9, 2025, 4:55pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/3 "2025-01-09T16:55:04Z")
Thanks for the info .I will try to modify the config to support curator v8 . ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [January 9, 2025, 5:16pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/4 "2025-01-09T17:16:43Z")
I have tried with new config as below , ```auto --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" elasticsearch: client: hosts: - http://127.0.0.1:9301 cloud_id: ca_certs: client_cert: client_key: verify_certs: request_timeout: 30 other_settings: master_only: False username: password: api_key: id: api_key: token: logging: loglevel: DEBUG logfile: /root/curator/logs/curator.log logformat: default blacklist: ['elasticsearch', 'urllib3'] ``` I got the below error in the log file: ```auto 2025-01-09 17:02:17,362 INFO elastic_transport.transport perform_request:349 HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.08 [status:404 duration:0.003s] 2025-01-09 17:02:17,366 INFO elastic_transport.transport perform_request:349 HEAD http://127.0.0.1:9301/_alias/heartbeat-7.12.0-2025.01.03-001292 [status:404 duration:0.003s] 2025-01-09 17:02:17,369 INFO elastic_transport.transport perform_request:349 HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.07 [status:404 duration:0.003s] 2025-01-09 17:02:17,372 INFO elastic_transport.transport perform_request:349 HEAD http://127.0.0.1:9301/_alias/oracle-2022.03.06 [status:404 duration:0.003s] 2025-01-09 17:02:17,390 INFO elastic_transport.transport perform_request:349 GET http://127.0.0.1:9301/.geoip_databases,.kibana_task_manager_7.17.16_001,.monitoring-alerts-7,.monitoring-es-7-2025.01.03,.monitoring-es-7-2025.01.04,.monitoring-es-7-2025.01.05,.monitoring-es-7-2025.01.06,.monitoring-es-7-2025.01.07,.monitoring-es-7-2025.01.08,.monitoring-es-7-2025.01.09,.monitoring-kibana-7-2025.01.08,.monitoring-kibana-7-2025.01.09,.reporting-2022-04-03,.reporting-2024-03-17,.tasks,access-%5B2024.12.09%5D,access-%5B2024.12.30%5D,access-%5B2024.12.31%5D,access-%5B2025.01.09%5D,api_access-%5B2024.12.28%5D,api_access-%5B2024.12.29%5D,api_access-%5B2024.12.30%5D,api_access-%5B2025.01.01%5D,api_access-%5B2025.01.02%5D,api_access-%5B2025.01.03%5D,api_access-%5B2025.01.04%5D,heartbeat-7.12.0-2024.12.31-001289,heartbeat-7.12.0-2025.01.03-001292,heartbeat-7.12.0-2025.01.05-001294,heartbeat-7.12.0-2025.01.06-001295,heartbeat-7.12.0-2025.01.08-001297,ilm-history-2-000010,ohs-%5B2024.12.20%5D,ohs-%5B2024.12.21%5D,ohs-%5B2024.12.22%5D,ohs-%5B2024.12.23%5D,ohs-%5B2024.12.24%5D,ohs-%5B2024.12.25%5D,ohs-%5B2024.12.26%5D,ohs-%5B2024.12.31%5D,ohs-%5B2025.01.05%5D,ohs-%5B2025.01.06%5D,ohs-%5B2025.01.07%5D,ohs-%5B2025.01.08%5D,ohs-%5B2025.01.09%5D,oracle-2021.10.20,oracle-2021.10.21,oracle-2021.10.22,oracle-2021.10.23,oracle-2021.10.24,oracle-2021.10.25,oracle-2021.10.26,oracle-2021.10.27,oracle-2021.10.28,oracle-2021.10.29,oracle-2022.01.20,oracle-2022.01.21,oracle-2022.01.22,oracle-2022.01.23,oracle-2022.01.24,oracle-2022.01.25,oracle-2022.01.26,oracle-2022.01.27,oracle-2022.01.28,oracle-2022.01.29,oracle-2022.01.30,oracle-2022.01.31,oracle-2022.02.01,oracle-2022.02.02,oracle-2022.02.03,oracle-2022.02.04,oracle-2022.02.05,oracle-2022.02.06,oracle-2022.02.07,oracle-2022.02.08,oracle-2022.02.09,oracle-2022.02.10,oracle-2022.02.11,oracle-2022.02.12,oracle-2022.02.13,oracle-2022.02.14,oracle-2022.02.15,oracle-2022.02.16,oracle-2022.02.17,oracle-2022.02.18,oracle-2022.02.19,oracle-2022.02.27,oracle-2022.02.28,oracle-2022.03.01,oracle-2022.03.02,oracle-2022.03.03,oracle-2022.03.04,oracle-2022.03.05,oracle-2022.03.06,oracle-2022.03.07,oracle-2022.03.08,oracle-2022.03.09,oracle-2022.03.10,oracle-2022.03.11,oracle-2022.03.12,oracle-2022.03.13,oracle-2022.03.14,oracle-2022.03.15,oracle-2022.03.16,oracle-2022.03.17,oracle-2022.03.18,oracle-2022.03.19,oracle-2022.03.20,oracle-2022.03.21,oracle-2022.03.22,oracle-2022.03.23,oracle-2022.03.24,oracle-2022.03.25,oracle-2022.03.26,oracle-2022.03.27,oracle-2022.03.30,oracle-2022.03.31,oracle-2022.04.01,oracle-2022.04.02,oracle-2022.04.03,oracle-2022.04.04,oracle-2022.04.05,oracle-2022.04.06,oracle-2022.04.07,oracle-2022.04.08,ords-%5B2024.12.23%5D,ords-%5B2024.12.24%5D,ords-%5B2024.12.25%5D,ords-%5B2024.12.26%5D,ords-%5B2024.12.30%5D,ords-%5B2024.12.31%5D,ords-%5B2025.01.05%5D,ords-%5B2025.01.06%5D,ords-%5B2025.01.07%5D,ords-%5B2025.01.08%5D,st-active-users-%5B2024.12.24%5D,st-active-users-%5B2024.12.30%5D,st-active-users-%5B2024.12.31%5D,st-active-users-%5B2025.01.04%5D,st-active-users-%5B2025.01.05%5D,st-active-users-%5B2025.01.06%5D,st-active-users-%5B2025.01.07%5D,st-active-users-%5B2025.01.08%5D,st-active-users-%5B2025.01.09%5D/_settings [status:400 duration:0.017s] 2025-01-09 17:02:17,390 ERROR curator.cli exception_handler:91 Failed to complete action: delete_indices. : BadRequestError(400, 'illegal_argument_exception', 'Indices [.geoip_databases] use and access is reserved for system operations') ``` FYI - Please find below for my existing config ```auto --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" client: hosts: - node1.xxxx.com - node02.xxxx.com - node03.xxxx.com port: 9301 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: DEBUG logfile: /root/curator/logs/curator.log logformat: default blacklist: ['elasticsearch', 'urllib3'] ``` Could you please check and let me know? Thanks for your support. Thanks. ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [January 10, 2025, 6:47am UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/5 "2025-01-10T06:47:04Z")
While running the curator manually i am getting the below warning. root@logsdev:~/curator# ./run.sh ```auto /usr/local/lib/python3.8/dist-packages/curator/indexlist.py:149: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .async-search, .kibana_7.12.0_001, .kibana_7.17.16_001, .reporting-2023-02-12, .reporting-2024-01-07, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default return self.client.indices.get_settings(index=to_csv(data)) ``` root@logsdev:~/curator# cat run.sh ```auto #!/usr/bin/env bash curator --config /root/curator/curator.config.yml /root/curator/actions.yml ``` Please find below my actions.yml file. ## root@logsdev:~/curator# cat actions.yml ```auto -- # # We don't want the disk to fill up too much so use the following action set to clear down # unnecessary data. # # You'll need to change this or add to it if you ever do rollover indexes instead. # actions: 1: action: delete_indices description: "Delete indices older than 31 days" options: ignore_empty_list: True disable_action: False filters: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 31 ``` root@logsdev:~/curator# could you please check ? Thanks, Padma. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [January 10, 2025, 4:46pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/6 "2025-01-10T16:46:17Z")
> `Indices [.geoip_databases] use and access is reserved for system operations` First, you appear to have only a single `age` filter. Granted, this will look for a `####.##.##` pattern in indices and try to match that to `Year.Month.Day` numbers for age calculation, but it is searching all indices first. This is causing Curator to evaluate system indices, i.e. those that are preceded by dots/periods. Since you also appear to be using a newer version of Elasticsearch that is halting operations against protected system indices, you must exclude them from consideration to avoid this 400 error. There are two ways to exclude system indices. You can use either or both, but they must be used for _each_ defined `action`. **Exclude all dot prefixed indices** Make your first `filter` a regex like this: ```yaml - filtertype: pattern kind: regex exclude: true value: "^\\.*" ``` You can add as many filters as you want, but they will always be processed in the order in which they appear in the YAML file. **Use a `search_pattern` (now called [multi-target syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index))** You can use an Elasticsearch [multi-target syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index) to include/exclude indices this way: ```yaml --- actions: 1: action: delete_indices description: "Example of search_pattern" options: search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*' ``` This search pattern begins by including all (`*`) indices, but then explicitly excludes (`-`) `.geoip_databases`, anything starting with `.fleet`, and anything starting with `.kibana`. You must start with _some_ indices before you can exclude. In this example, the wildcard for all is used first, and then the exclusions are listed. ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [January 14, 2025, 11:13am UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/7 "2025-01-14T11:13:39Z")
It's working now .but i am seeing the below on the log file. ```auto on': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None} 2025-01-14 10:56:17,817 DEBUG curator.indexlist iterate_filters:1466 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None} 2025-01-14 10:56:17,817 DEBUG curator.indexlist iterate_filters:1472 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None} 2025-01-14 10:56:17,818 DEBUG curator.indexlist iterate_filters:1473 Pre-instance: [] 2025-01-14 10:56:17,818 DEBUG curator.indexlist filter_by_age:683 Filtering indices by age 2025-01-14 10:56:17,818 DEBUG curator.indexlist get_index_settings:328 Getting index settings -- BEGIN 2025-01-14 10:56:17,818 DEBUG curator.indexlist empty_list_check:448 Checking for empty list 2025-01-14 10:56:17,818 INFO curator.cli exception_handler:78 Skipping action "delete_indices" due to empty list: 2025-01-14 10:56:17,818 INFO curator.cli run:236 Action ID: 1, "delete_indices" completed. 2025-01-14 10:56:17,818 INFO curator.cli run:237 All actions completed. ``` Please find the below my config files: ```auto root@logsdev:~/curator# cat actions.yml --- # # We don't want the disk to fill up too much so use the following action set to clear down # unnecessary data. # # You'll need to change this or add to it if you ever do rollover indexes instead. # actions: 1: action: delete_indices description: "Delete indices older than 31 days" options: allow_ilm_indices: true ignore_empty_list: true disable_action: false filters: - filtertype: pattern kind: prefix value: '.' # Exclude indices starting with a dot (system indices) exclude: true - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 31 ``` curator.config.yml ```auto root@logsdev:~/curator# cat curator.config.yml --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" elasticsearch: client: hosts: - http://127.0.0.1:9301 - http://logsnode02.corehr.com:9301 - http://logsnode03.corehr.com:9301 cloud_id: ca_certs: client_cert: client_key: verify_certs: request_timeout: 30 other_settings: master_only: False username: password: api_key: id: api_key: token: logging: loglevel: DEBUG logfile: /root/curator/logs/curator.log logformat: default blacklist: ['elastic_transport', 'urllib3'] ``` I dont have the logs older than 31 days in the indices .Due to that i am getting the empty list. Please confirm. Thanks. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [January 14, 2025, 2:12pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/8 "2025-01-14T14:12:09Z")
> [@padmavathi](#): > > ```auto > - filtertype: pattern > kind: prefix > value: '.' # Exclude indices starting with a dot (system indices) > exclude: true > > ``` This is not what I said to use. The `prefix` pattern is still a regular expression. It just adds the `^` anchor before whatever you put in `value`. A plain `.` will match any character. This is why all of your indices are being matched and excluded, resulting the empty list error. What I shared and expect you to use is: ```auto - filtertype: pattern kind: regex exclude: true value: "^\\.*" ``` This is a pure regular expression that includes the `^` anchor to match the beginning of the line. It then escapes the period `\\` which says to match an actual period and not just any character. Replace the incorrect pattern filter with the one I already provided and everything will work. ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [January 14, 2025, 2:30pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/9 "2025-01-14T14:30:12Z")
Thank you . I have updated my config file as below .but still i am getting the same log file. please find below my config: ```auto root@logsdev:~/curator# cat actions.yml --- # # We don't want the disk to fill up too much so use the following action set to clear down # unnecessary data. # # You'll need to change this or add to it if you ever do rollover indexes instead. # actions: 1: action: delete_indices description: "Delete indices older than 31 days" options: allow_ilm_indices: true ignore_empty_list: true disable_action: false filters: - filtertype: pattern kind: regex exclude: true value: "^\\.*" - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 31 ``` Output log file: ```auto 2025-01-14 14:23:33,799 DEBUG curator.indexlist __not_actionable:58 Index st-active-users-[2025.01.04] is not actionable, removing from list. 2025-01-14 14:23:33,799 DEBUG curator.indexlist filter_by_regex:637 Filter by regex: Index: oracle-2022.03.09 2025-01-14 14:23:33,799 DEBUG curator.indexlist __not_actionable:58 Index oracle-2022.03.09 is not actionable, removing from list. 2025-01-14 14:23:33,799 DEBUG curator.indexlist filter_by_regex:637 Filter by regex: Index: oracle-2022.03.08 2025-01-14 14:23:33,799 DEBUG curator.indexlist __not_actionable:58 Index oracle-2022.03.08 is not actionable, removing from list. 2025-01-14 14:23:33,799 DEBUG curator.indexlist filter_by_regex:637 Filter by regex: Index: oracle-2022.03.07 2025-01-14 14:23:33,799 DEBUG curator.indexlist __not_actionable:58 Index oracle-2022.03.07 is not actionable, removing from list. 2025-01-14 14:23:33,799 DEBUG curator.indexlist filter_by_regex:637 Filter by regex: Index: oracle-2022.03.06 2025-01-14 14:23:33,799 DEBUG curator.indexlist __not_actionable:58 Index oracle-2022.03.06 is not actionable, removing from list. 2025-01-14 14:23:33,799 DEBUG curator.indexlist iterate_filters:1475 Post-instance: [] 2025-01-14 14:23:33,799 DEBUG curator.indexlist iterate_filters:1459 Top of the loop: [] 2025-01-14 14:23:33,799 DEBUG curator.indexlist iterate_filters:1460 Un-parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2025-01-14 14:23:33,800 DEBUG es_client.helpers.schemacheck __init__ :60 Schema: {'aliases': Any(, , msg=None), 'allocation_type': Any(, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, , msg=None), 'date_from_format': Any(None, , msg=None), 'date_to': Any(None, , msg=None), 'date_to_format': Any(None, , msg=None), 'direction': Any(, msg=None), 'disk_space': , 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, , , , msg=None), 'field': Any(None, , msg=None), 'intersect': Any(None, , , , msg=None), 'key': Any(, msg=None), 'kind': Any(, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(, msg=None), 'period_type': Any(, msg=None), 'reverse': Any(None, , , , msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(, msg=None), 'size_behavior': Any(, msg=None), 'size_threshold': Any(Coerce(float, msg=None), msg=None), 'source': Any(, msg=None), 'state': Any(, msg=None), 'stats_result': Any(None, , msg=None), 'timestring': Any(None, , msg=None), 'threshold_behavior': Any(, msg=None), 'unit': Any(, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(, msg=None), 'use_age': , 'value': Any(, , , , msg=None), 'week_starts_on': Any(None, , msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']")} 2025-01-14 14:23:33,800 DEBUG es_client.helpers.schemacheck __init__ :62 "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2025-01-14 14:23:33,800 DEBUG curator.indexlist iterate_filters:1466 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2025-01-14 14:23:33,800 DEBUG curator.indexlist iterate_filters:1472 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'epoch': None, 'stats_result': 'min_value', 'exclude': False} 2025-01-14 14:23:33,800 DEBUG curator.indexlist iterate_filters:1473 Pre-instance: [] 2025-01-14 14:23:33,801 DEBUG curator.indexlist filter_by_age:683 Filtering indices by age 2025-01-14 14:23:33,801 DEBUG curator.indexlist get_index_settings:328 Getting index settings -- BEGIN 2025-01-14 14:23:33,801 DEBUG curator.indexlist empty_list_check:448 Checking for empty list 2025-01-14 14:23:33,801 INFO curator.cli exception_handler:78 Skipping action "delete_indices" due to empty list: 2025-01-14 14:23:33,801 INFO curator.cli run:236 Action ID: 1, "delete_indices" completed. 2025-01-14 14:23:33,801 INFO curator.cli run:237 All actions completed. Could you please check on this? Regards, Padma. ``` ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [January 21, 2025, 4:09pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/10 "2025-01-21T16:09:26Z")
Sorry for the delay. I was out of office. I cannot see enough of what happened here. Something seems amiss, however, as it is filtering _all_ indices and not just ones starting with a `.` I'm not sure why it's not working as expected. Please try changing: `value: "^\\.*"` to `value: "^\.*"` And add the `--dry-run` flag to your command-line so it shows what is being filtered, but does not delete anything yet. ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [January 30, 2025, 6:03pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/11 "2025-01-30T18:03:48Z")
No ,It didnt work.i am getting the same schema error: Schema error: extra keys not allowed @ data['elasticsearch']. It was not deleting the older indices . ```auto n': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None} 2025-01-30 17:45:05,545 DEBUG es_client.helpers.schemacheck __init__ :60 Schema: {'aliases': Any(, , msg=None), 'allocation_type': Any(, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, , msg=None), 'date_from_format': Any(None, , msg=None), 'date_to': Any(None, , msg=None), 'date_to_format': Any(None, , msg=None), 'direction': Any(, msg=None), 'disk_space': , 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, , , , msg=None), 'field': Any(None, , msg=None), 'intersect': Any(None, , , , msg=None), 'key': Any(, msg=None), 'kind': Any(, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(, msg=None), 'period_type': Any(, msg=None), 'reverse': Any(None, , , , msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(, msg=None), 'size_behavior': Any(, msg=None), 'size_threshold': Any(Coerce(float, msg=None), msg=None), 'source': Any(, msg=None), 'state': Any(, msg=None), 'stats_result': Any(None, , msg=None), 'timestring': Any(None, , msg=None), 'threshold_behavior': Any(, msg=None), 'unit': Any(, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(, msg=None), 'use_age': , 'value': Any(, , , , msg=None), 'week_starts_on': Any(None, , msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']")} 2025-01-30 17:45:05,545 DEBUG es_client.helpers.schemacheck __init__ :62 "filter" config: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None} 2025-01-30 17:45:05,545 DEBUG curator.indexlist iterate_filters:1466 Parsed filter args: {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None} 2025-01-30 17:45:05,546 DEBUG curator.indexlist iterate_filters:1472 Filter args: {'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 31, 'stats_result': 'min_value', 'exclude': False, 'epoch': None} 2025-01-30 17:45:05,546 DEBUG curator.indexlist iterate_filters:1473 Pre-instance: [] 2025-01-30 17:45:05,546 DEBUG curator.indexlist filter_by_age:683 Filtering indices by age 2025-01-30 17:45:05,546 DEBUG curator.indexlist get_index_settings:328 Getting index settings -- BEGIN 2025-01-30 17:45:05,546 DEBUG curator.indexlist empty_list_check:448 Checking for empty list 2025-01-30 17:45:05,546 INFO curator.cli exception_handler:78 Skipping action "delete_indices" due to empty list: 2025-01-30 17:45:05,546 INFO curator.cli run:236 Action ID: 1, "delete_indices" completed. 2025-01-30 17:45:05,546 INFO curator.cli run:237 All actions completed. Schema error: extra keys not allowed @ data['elasticsearch'] Schema error: extra keys not allowed @ data['elasticsearch'] ``` Please find the attached config files ```auto root@logsdev:~/curator# cat curator.config.yml --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" elasticsearch: client: hosts: - http://127.0.0.1:9301 - http://logsnode02.corehr.com:9301 - http://logsnode03.corehr.com:9301 cloud_id: ca_certs: client_cert: client_key: verify_certs: request_timeout: 30 other_settings: master_only: False username: password: api_key: id: api_key: token: logging: loglevel: DEBUG logfile: /root/curator/logs/curator.log logformat: default blacklist: ['elastic_transport', 'urllib3'] action file ``` ```auto root@logsdev:~/curator# cat actions.yml --- # # We don't want the disk to fill up too much so use the following action set to clear down # unnecessary data. # # You'll need to change this or add to it if you ever do rollover indexes instead. # actions: 1: action: delete_indices description: "Delete indices older than 31 days" options: allow_ilm_indices: true ignore_empty_list: true disable_action: false filters: - filtertype: pattern kind: regex exclude: true value: "^\\.*" - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 31 ``` could you please check on this ? Thanks. ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [February 5, 2025, 12:06am UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/12 "2025-02-05T00:06:06Z")
I think ,curator version 8.0.17 is not compatible with ES version 8. [Curator and Elasticsearch 8 - Elastic Stack / Elasticsearch - Discuss the Elastic Stack](https://discuss.elastic.co/t/curator-and-elasticsearch-8/316550) Can we go with ILM policy ? Please confirm . Thanks. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [February 5, 2025, 5:34pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/13 "2025-02-05T17:34:01Z")
The post you linked is dated October 2022. Curator 8.0.0 was released [January 31, 2023](https://github.com/elastic/curator/releases/tag/v8.0.0). ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [February 5, 2025, 6:25pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/14 "2025-02-05T18:25:46Z")
I am facing issue with the age filter .its not removing the older indices and also im facing the schema error especially with the cron jobs . I am using curator version 8.0.17 and ES version 8.0.1. Thanks. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [February 5, 2025, 6:45pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/15 "2025-02-05T18:45:14Z")
I really do not know why the `value: "^\\.*"` pattern isn't working for you: ```yaml actions: 1: description: "This action should not affect anything. It's here to show filter behavior" action: open options: allow_ilm_indices: True filters: - filtertype: pattern kind: regex exclude: true value: "^\\.*" ``` Sample of debug log output from my own cluster running 8.17.0: ```auto 2025-02-05 11:04:42,769 DEBUG curator.indexlist iterate_filters:1452 Iterating over a list of filters 2025-02-05 11:04:42,769 DEBUG curator.indexlist iterate_filters:1457 All filters: [{'filtertype': 'pattern', 'kind': 'regex', 'exclude': True, 'value': '^\\.*'}] 2025-02-05 11:04:42,773 DEBUG curator.indexlist filter_by_regex:620 Filtering indices by regex 2025-02-05 11:04:42,773 DEBUG curator.indexlist empty_list_check:448 Checking for empty list 2025-02-05 11:04:42,773 DEBUG curator.indexlist working_list:459 Generating working list of indices 2025-02-05 11:04:42,773 DEBUG curator.indexlist filter_by_regex:637 Filter by regex: Index: .ds-metrics-kubernetes.node-default-2024.11.24-000002 2025-02-05 11:04:42,773 DEBUG curator.indexlist __not_actionable:58 Index .ds-metrics-kubernetes.node-default-2024.11.24-000002 is not actionable, removing from list. 2025-02-05 11:04:42,773 DEBUG curator.indexlist filter_by_regex:637 Filter by regex: Index: .ds-metrics-kubernetes.state_daemonset-k8s-2025.02.04-000004 2025-02-05 11:04:42,773 DEBUG curator.indexlist __not_actionable:58 Index .ds-metrics-kubernetes.state_daemonset-k8s-2025.02.04-000004 is not actionable, removing from list. ... ``` There's a lot more of the same that follows. As configured, this is filtering indices that start with a `.`. Let's go back to my earlier recommendation to use a `search_pattern`. Since we're having some issues with the regex pattern not filtering indices that start with a `.` we will take the other recommended route. > [@theuntergeek](#): > > **Use a `search_pattern` (now called [multi-target syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index))** > You can use an Elasticsearch [multi-target syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#api-multi-index) to include/exclude indices this way: > > ```auto > --- > actions: > 1: > action: delete_indices > description: "Example of search_pattern" > options: > search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*' > > ``` > > This search pattern begins by including all (`*`) indices, but then explicitly excludes (`-`) `.geoip_databases`, anything starting with `.fleet`, and anything starting with `.kibana`. You must start with _some_ indices before you can exclude. In this example, the wildcard for all is used first, and then the exclusions are listed. Initially, you shared a warning level log message showing system indices that should not be accessed directly: ```auto /usr/local/lib/python3.8/dist-packages/curator/indexlist.py:149: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .async-search, .kibana_7.12.0_001, .kibana_7.17.16_001, .reporting-2023-02-12, .reporting-2024-01-07, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default return self.client.indices.get_settings(index=to_csv(data)) ``` This is only a warning, so it would not halt execution. But we should be in the habit of heeding these warnings and not directly accessing the named indices. Additionally, the `.geoip_databases` index and any other system index should also be omitted from consideration, including `.fleet*`, `.kibana*`, `.apm-agent-configuration`, `.async-search`, `.reporting-*`, `.triggered_watches`, and `.watches`. As such, I have taken the liberty of creating a `search_pattern` that includes all indices _except_ the names and patterns preceded by a `-` (negating the pattern/name): ```yaml search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*,-.apm-agent-configuration,-.async-search,-.reporting-*,-.triggered_watches,-.watches' ``` Please make your `actions.yml` file look like this: ```yaml --- actions: 1: action: delete_indices description: "Delete indices older than 31 days" options: allow_ilm_indices: true ignore_empty_list: true disable_action: false search_pattern: '*,-.geoip_databases,-.fleet*,-.kibana*,-.apm-agent-configuration,-.async-search,-.reporting-*,-.triggered_watches,-.watches' filters: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 31 ``` I have tested this with the `--dry-run` flag in my own cluster and it does match indices by `age` with this pattern as expected. ## Using `search_pattern` effectively Running a single action is simple and easy, and if you don't have too many indices it runs rather quickly. But keep in mind that with an `age` filter, Curator grabs metadata for all indices found by `search_pattern` before proceeding to the filters. On my own cluster, 529 indices were found by this `search_pattern`, and it took over 19 seconds to run, generating 2065 log lines in DEBUG mode. Curator did the work to gather the metadata on all 529 indices before filtering by age. But if I only needed to operate on a subset of all indices? Or what if I wanted my logs to be much shorter? By updating `search_pattern` to only gather indices starting with `.ds-metrics-kubernetes*`, I can simplify the `search_pattern`, shorten execution time, and reduce the log output size: ```yaml --- actions: 1: action: delete_indices description: "Delete indices older than 31 days" options: allow_ilm_indices: true ignore_empty_list: true disable_action: false search_pattern: '.ds-metrics-kubernetes*' filters: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 31 ``` In my cluster, this shortened the list of indices needing metadata to 60, and execution time dropped to 2.6 seconds, generating only 284 log lines in DEBUG mode. You could just as easily expand the `search_pattern` to include more indices you know you will be working on: ```yaml search_pattern: '.ds-metrics-kubernetes*,.ds-metrics-system*' ``` The index count climbed to 82, execution time was 3.6s, and log line count was 371. Please use this feature to your advantage! ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [February 5, 2025, 6:49pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/16 "2025-02-05T18:49:05Z")
Your schema error is caused by the last line: > [@padmavathi](#): > > ```auto > --- > # Remember, leave a key empty if there is no value. None will be a string, > # not a Python "NoneType" > elasticsearch: > client: > hosts: > - http://127.0.0.1:9301 > - http://logsnode02.corehr.com:9301 > - http://logsnode03.corehr.com:9301 > cloud_id: > ca_certs: > client_cert: > client_key: > verify_certs: > request_timeout: 30 > other_settings: > master_only: False > username: > password: > api_key: > id: > api_key: > token: > logging: > loglevel: DEBUG > logfile: /root/curator/logs/curator.log > logformat: default > blacklist: ['elastic_transport', 'urllib3'] > > action file > > ``` `action file` should not be here. That is the `extra key` that should not be there. ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [February 6, 2025, 12:03pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/17 "2025-02-06T12:03:37Z")
Thanks for the info . I have added the search pattern and i could able to deleted the older indices. please find below my action file: ```auto --- actions: 1: action: delete_indices description: "Delete indices older than 31 days" options: allow_ilm_indices: true ignore_empty_list: true disable_action: false search_pattern: '*,-.geoip_databases,-.apm-custom-link*,-.fleet*,-.tasks*,-.kibana*,-.monitoring-alerts-7*,-.reporting*,-.apm-agent-configuration,-.async-search,-.reporting-*,-.triggered_watches,-.watches' filters: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 31 ``` It's working fine manually . I have scheduled cronjob to run this task: `* * * * * /root/curator/run.sh >> /root/curator/logs/curator.log 2>&1` if i am rununing via cronjob im getting the same schema error . ```auto 2025-02-06 11:17:48,803 DEBUG curator.indexlist __excludify:77 Removed from actionable list: Index "st-active-users-[2025.01.28]" age (1738022400), direction: "older", point of reference, (1736162268) 2025-02-06 11:17:48,803 DEBUG curator.indexlist iterate_filters:1475 Post-instance: [] 2025-02-06 11:17:48,803 DEBUG curator.cli process_action:168 Pre Instantiation Action kwargs: {'master_timeout': 30} 2025-02-06 11:17:48,803 DEBUG curator.actions.delete_indices __init__ :32 master_timeout value: 30s 2025-02-06 11:17:48,803 DEBUG curator.cli process_action:174 Doing the action here. 2025-02-06 11:17:48,803 DEBUG curator.indexlist empty_list_check:448 Checking for empty list 2025-02-06 11:17:48,803 INFO curator.cli exception_handler:78 Skipping action "delete_indices" due to empty list: 2025-02-06 11:17:48,803 INFO curator.cli run:236 Action ID: 1, "delete_indices" completed. 2025-02-06 11:17:48,803 INFO curator.cli run:237 All actions completed. Schema error: extra keys not allowed @ data['elasticsearch'] Schema error: extra keys not allowed @ data['elasticsearch'] Schema error: extra keys not allowed @ data['elasticsearch'] Schema error: extra keys not allowed @ data['elasticsearch'] Schema error: extra keys not allowed @ data['elasticsearch'] Schema error: extra keys not allowed @ data['elasticsearch'] Schema error: extra keys not allowed @ data['elasticsearch'] Schema error: extra keys not allowed @ data['elasticsearch'] ``` Could you please advise on this ? Thanks. ---
**Author:** ![padmavathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/padmavathi/32/134714_2.png) [@padmavathi](https://discuss.elastic.co/u/padmavathi)\ **Post date:** [February 7, 2025, 1:53pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/18 "2025-02-07T13:53:23Z")
The issue is fixed now .Thank you . ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 7, 2025, 1:53pm UTC](https://discuss.elastic.co/t/issues-when-running-curator-8-0-17/372996/19 "2025-03-07T13:53:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.