# Using Curator **URL:** \ **Category:** Elasticsearch\ **Created:** [March 25, 2020, 5:06pm UTC](https://discuss.elastic.co/t/using-curator/225032 "2020-03-25T17:06:29Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Bruceclegg](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@Bruceclegg](https://discuss.elastic.co/u/Bruceclegg)\ **Post date:** [March 25, 2020, 5:06pm UTC](https://discuss.elastic.co/t/using-curator/225032/1 "2020-03-25T17:06:29Z")
I've been tasked with setting up Curator to delete data more than 40 days old on our existing ELK 6.8.6 server. I did not set up elk and am pretty much unfamiliar with how it is configured. I set up the yum repository, installed the latest curator. I set up a (default) config.yml and action.yml. ... config file: \*# cat /etc/curator/config.yml \* _client:_ - hosts:\* - - 127.0.0.1\* - port: 9200\* - url\_prefix:\* - use\_ssl: False\* - certificate:\* - client\_cert:\* - client\_key:\* - ssl\_no\_validate: False\* - http\_auth: \* - timeout: 30\* - master\_only: False\* _logging:_ - loglevel: DEBUG\* - logfile: /home/bclegg/testlog\* - logformat: default\* - blacklist: ['elasticsearch', 'urllib3']\* _..._ action file: ... # cat /etc/curator/action.yml _actions:_ - 1:\* - action: close\* - description: \>-\* - ``` Close indices older than 40 days (based on index name).* ``` - options:\* - ``` ignore_empty_list: True* ``` - ``` delete_aliases: False* ``` - ``` disable_action: False* ``` - filters:\* - - filtertype: pattern\* - ``` kind: prefix* ``` - ``` value: nginx-* ``` - - filtertype: age\* - ``` source: name* ``` - ``` direction: older* ``` - ``` timestring: '%Y.%m.%d'* ``` - ``` unit: days* ``` - ``` unit_count: 40* ``` - 2:\* - action: delete\_indices\* - description: \>-\* - ``` Delete indices older than 40 days (based on index name).* ``` - options:\* - ``` ignore_empty_list: True* ``` - ``` disable_action: False* ``` - filters:\* - - filtertype: pattern\* - ``` kind: prefix* ``` - ``` value: nginx-* ``` - - filtertype: age\* - ``` source: name* ``` - ``` direction: older* ``` - ``` timestring: '%Y.%m.%d'* ``` - ``` unit: days* ``` - ``` unit_count: 40* ``` ... fwiw, I chose nginx as a place to start. Once I get this to work, I'll expand. When I attempt a dry run, I get: # /usr/bin/curator --dry-run /etc/curator/action.yml ... _2020-03-25 12:56:40,839 DEBUG curator.cli run:128 action\_disabled = False_ _2020-03-25 12:56:40,839 DEBUG curator.cli run:132 continue\_if\_exception = False_ _2020-03-25 12:56:40,839 DEBUG curator.cli run:134 timeout\_override = 180_ _2020-03-25 12:56:40,839 DEBUG curator.cli run:136 ignore\_empty\_list = True_ _2020-03-25 12:56:40,839 DEBUG curator.cli run:138 allow\_ilm\_indices = False_ _2020-03-25 12:56:40,839 INFO curator.cli run:148 Preparing Action ID: 1, "close"_ _2020-03-25 12:56:40,839 INFO curator.cli run:162 Creating client object and testing connection_ _2020-03-25 12:56:40,839 DEBUG curator.utils get\_client:809 kwargs = {'hosts': ['127.0.0.1'], 'port': 9200, 'use\_ssl': False, 'ssl\_no\_validate': False, 'master\_only': False, 'url\_prefix': '', 'aws\_token': None, 'aws\_key': None, 'http\_auth': None, 'client\_key': None, 'client\_cert': None, 'aws\_secret\_key': None, 'certificate': None, 'aws\_sign\_request': False, 'timeout': 180}_ _2020-03-25 12:56:40,840 DEBUG curator.utils get\_client:871 Checking for AWS settings_ _2020-03-25 12:56:40,844 DEBUG curator.utils get\_client:886 "requests\_aws4auth" module present, but not used._ _2020-03-25 12:56:40,844 INFO curator.utils get\_client:903 Instantiating client object_ _2020-03-25 12:56:40,844 INFO curator.utils get\_client:906 Testing client connectivity_ _2020-03-25 12:56:40,848 ERROR curator.utils get\_client:915 HTTP 401 error: ^M_ _401 Authorization Required^M_ _^M_ # _401 Authorization Required_ _^M_ _* * * nginx/1.17.5^M_ _^M_ _^M_ ... So the error I'm getting is Authorization Required. How do I provide authentication? I've tried a few things with no luck. Do I need to provide certs? Which certs? The elasticsearch certs from /etc/elasticsearch/config/certs? Any help is very much appreciated. ---
**Author:** ![Bruceclegg](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@Bruceclegg](https://discuss.elastic.co/u/Bruceclegg)\ **Post date:** [March 25, 2020, 5:38pm UTC](https://discuss.elastic.co/t/using-curator/225032/2 "2020-03-25T17:38:01Z")
I tried pointing to the elasticsearch certs - now I'm seeing this in the log: _"/etc/curator/config.yml" 19L, 407C written_ _2020-03-25 13:35:09,271 DEBUG curator.cli run:128 action\_disabled = False_ _2020-03-25 13:35:09,271 DEBUG curator.cli run:132 continue\_if\_exception = False_ _2020-03-25 13:35:09,271 DEBUG curator.cli run:134 timeout\_override = 180_ _2020-03-25 13:35:09,271 DEBUG curator.cli run:136 ignore\_empty\_list = True_ _2020-03-25 13:35:09,271 DEBUG curator.cli run:138 allow\_ilm\_indices = False_ _2020-03-25 13:35:09,271 INFO curator.cli run:148 Preparing Action ID: 1, "close"_ _2020-03-25 13:35:09,271 INFO curator.cli run:162 Creating client object and testing connection_ _2020-03-25 13:35:09,271 DEBUG curator.utils get\_client:809 kwargs = {'hosts': ['127.0.0.1'], 'port': 9200, 'use\_ssl': True, 'client\_cert': '/etc/elasticsearch/config/certs/elk/elk.crt', 'client\_key': '/etc/elasticsearch/config/certs/elk/elk.key', 'ssl\_no\_validate': False, 'master\_only': False, 'url\_prefix': '', 'aws\_token': None, 'certificate': None, 'aws\_secret\_key': None, 'http\_auth': None, 'aws\_key': None, 'aws\_sign\_request': False, 'timeout': 180}_ _2020-03-25 13:35:09,271 DEBUG curator.utils get\_client:815 Attempting to verify SSL certificate._ _2020-03-25 13:35:09,272 DEBUG curator.utils get\_client:871 Checking for AWS settings_ _2020-03-25 13:35:09,276 DEBUG curator.utils get\_client:886 "requests\_aws4auth" module present, but not used._ _2020-03-25 13:35:09,276 INFO curator.utils get\_client:903 Instantiating client object_ _2020-03-25 13:35:09,276 INFO curator.utils get\_client:906 Testing client connectivity_ _2020-03-25 13:35:09,403 ERROR curator.utils get\_client:915 HTTP N/A error: HTTPSConnectionPool(host='127.0.0.1', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG\_VERSION\_NUMBER] wrong version number (\_ssl.c:1076)')))_ _2020-03-25 13:35:09,404 CRITICAL curator.utils get\_client:923 Curator cannot proceed. Exiting._ ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [March 25, 2020, 5:51pm UTC](https://discuss.elastic.co/t/using-curator/225032/3 "2020-03-25T17:51:28Z")
So, it appears that you were able to connect to the cluster without certs before, it merely needed a username and password. You should define [`http_auth`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/configfile.html#http_auth) to set username and password. ---
**Author:** ![Bruceclegg](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@Bruceclegg](https://discuss.elastic.co/u/Bruceclegg)\ **Post date:** [March 25, 2020, 8:09pm UTC](https://discuss.elastic.co/t/using-curator/225032/4 "2020-03-25T20:09:16Z")
Thank you for your quick reply. I'm not sure I've ever been connected with curator. I think I have not. I've tried several more changes to the config.yml file here is my latest attempt with some of the information screened out: # /usr/bin/curator --dry-run /etc/curator/action.yml from the logfile: client: hosts: - 10.X.X.X port: 9200 url\_prefix: use\_ssl: False certificate: client\_cert: /etc/elasticsearch/config/certs/elk/elk.crt client\_key: /etc/elasticsearch/config/certs/elk/elk.key ssl\_no\_validate: False http\_auth: root: timeout: 30 master\_only: False logging: loglevel: DEBUG logfile: /home/bclegg/testlog logformat: default blacklist: ['elasticsearch', 'urllib3'] ~ ~ "/etc/curator/config.yml" 19L, 427C written 2020-03-25 15:55:39,407 DEBUG curator.cli run:110 Client and logging options validated. 2020-03-25 15:55:39,407 DEBUG curator.cli run:114 default\_timeout = 30 2020-03-25 15:55:39,408 DEBUG curator.cli run:118 action\_file: /etc/curator/action.yml 2020-03-25 15:55:39,420 DEBUG curator.cli run:120 action\_config: {'actions': {1: {'action': 'close', 'description': 'Close indices older than 40 days (based on index name).', 'options': {'ignore\_empty\_list': True, 'delete\_aliases': False, 'disable\_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit\_count': 40}]}, 2: {'action': 'delete\_indices', 'description': 'Delete indices older than 40 days (based on index name).', 'options': {'ignore\_empty\_list': True, 'disable\_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit\_count': 40}]}}} 2020-03-25 15:55:39,420 DEBUG curator.validators.SchemaCheck **init** :26 Schema: {'actions': \} 2020-03-25 15:55:39,420 DEBUG curator.validators.SchemaCheck **init** :27 "Actions File" config: {'actions': {1: {'action': 'close', 'description': 'Close indices older than 40 days (based on index name).', 'options': {'ignore\_empty\_list': True, 'delete\_aliases': False, 'disable\_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit\_count': 40}]}, 2: {'action': 'delete\_indices', 'description': 'Delete indices older than 40 days (based on index name).', 'options': {'ignore\_empty\_list': True, 'disable\_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit\_count': 40}]}}} 2020-03-25 15:55:39,421 DEBUG curator.validators.SchemaCheck **init** :26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster\_routing', 'create\_index', 'delete\_indices', 'delete\_snapshots', 'forcemerge', 'freeze', 'index\_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster\_routing', 'create\_index', 'delete\_indices', 'delete\_snapshots', 'forcemerge', 'freeze', 'index\_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']")} 2020-03-25 15:55:39,421 DEBUG curator.validators.SchemaCheck **init** :27 "action type" config: {'action': 'close', 'description': 'Close indices older than 40 days (based on index name).', 'options': {'ignore\_empty\_list': True, 'delete\_aliases': False, 'disable\_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit\_count': 40}]} @ @ @ @ @ @ @ @ "testlog" [readonly] 58L, 20033C 2020-03-25 15:55:39,439 DEBUG curator.cli run:128 action\_disabled = False 2020-03-25 15:55:39,439 DEBUG curator.cli run:132 continue\_if\_exception = False 2020-03-25 15:55:39,439 DEBUG curator.cli run:134 timeout\_override = 180 2020-03-25 15:55:39,439 DEBUG curator.cli run:136 ignore\_empty\_list = True 2020-03-25 15:55:39,439 DEBUG curator.cli run:138 allow\_ilm\_indices = False 2020-03-25 15:55:39,439 INFO curator.cli run:148 Preparing Action ID: 1, "close" 2020-03-25 15:55:39,440 INFO curator.cli run:162 Creating client object and testing connection 2020-03-25 15:55:39,440 DEBUG curator.utils get\_client:809 kwargs = {'hosts': ['10.X.X.X'], 'port': 9200, 'use\_ssl': False, 'client\_cert': '/etc/elasticsearch/config/certs/elk/elk.crt', 'client\_key': '/etc/elasticsearch/config/certs/elk/elk.key', 'ssl\_no\_validate': False, 'http\_auth': 'root:', 'master\_only': False, 'aws\_secret\_key': None, 'url\_prefix': '', 'aws\_key': None, 'certificate': None, 'aws\_token': None, 'aws\_sign\_request': False, 'timeout': 180} 2020-03-25 15:55:39,440 DEBUG curator.utils get\_client:871 Checking for AWS settings 2020-03-25 15:55:39,446 DEBUG curator.utils get\_client:886 "requests\_aws4auth" module present, but not used. 2020-03-25 15:55:39,446 INFO curator.utils get\_client:903 Instantiating client object 2020-03-25 15:55:39,447 INFO curator.utils get\_client:906 Testing client connectivity 2020-03-25 15:55:39,452 ERROR curator.utils get\_client:915 HTTP 401 error: ^M 401 Authorization Required^M ^M # 401 Authorization Required ^M * * * nginx/1.17.5^M ^M ^M 2020-03-25 15:55:39,452 CRITICAL curator.utils get\_client:923 Curator cannot proceed. Exiting. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [March 25, 2020, 8:43pm UTC](https://discuss.elastic.co/t/using-curator/225032/5 "2020-03-25T20:43:39Z")
> [@Bruceclegg](#): > > http\_auth: root: So, this is asking for Elasticsearch credentials, rather than operating system level credentials. Is there someone on your team who could help you get super-user credentials to Elasticsearch? ---
**Author:** ![Bruceclegg](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@Bruceclegg](https://discuss.elastic.co/u/Bruceclegg)\ **Post date:** [March 25, 2020, 8:56pm UTC](https://discuss.elastic.co/t/using-curator/225032/6 "2020-03-25T20:56:21Z")
I asked and all they had was a kibana user. I tried it and I was able process action.yml without any errors in the logfile. Thank You ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 22, 2020, 8:56pm UTC](https://discuss.elastic.co/t/using-curator/225032/7 "2020-04-22T20:56:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.