I've been tasked with setting up Curator to delete data more than 40 days old on our existing ELK 6.8.6 server. I did not set up elk and am pretty much unfamiliar with how it is configured.
I set up the yum repository, installed the latest curator. I set up a (default) config.yml and action.yml.
...
config file:
*# cat /etc/curator/config.yml *
client:
- hosts:*
-
- 127.0.0.1*
- port: 9200*
- url_prefix:*
- use_ssl: False*
- certificate:*
- client_cert:*
- client_key:*
- ssl_no_validate: False*
- http_auth: *
- timeout: 30*
- master_only: False*
logging:
- loglevel: DEBUG*
- logfile: /home/bclegg/testlog*
- logformat: default*
- blacklist: ['elasticsearch', 'urllib3']*
...
action file:
...
cat /etc/curator/action.yml
actions:
-
1:*
-
action: close*
-
description: >-*
-
Close indices older than 40 days (based on index name).* -
options:*
-
ignore_empty_list: True* -
delete_aliases: False* -
disable_action: False* -
filters:*
-
- filtertype: pattern*
-
kind: prefix* -
value: nginx-* -
- filtertype: age*
-
source: name* -
direction: older* -
timestring: '%Y.%m.%d'* -
unit: days* -
unit_count: 40* -
2:*
-
action: delete_indices*
-
description: >-*
-
Delete indices older than 40 days (based on index name).* -
options:*
-
ignore_empty_list: True* -
disable_action: False* -
filters:*
-
- filtertype: pattern*
-
kind: prefix* -
value: nginx-* -
- filtertype: age*
-
source: name* -
direction: older* -
timestring: '%Y.%m.%d'* -
unit: days* -
unit_count: 40*
...
fwiw, I chose nginx as a place to start. Once I get this to work, I'll expand.
When I attempt a dry run, I get:
/usr/bin/curator --dry-run /etc/curator/action.yml
...
2020-03-25 12:56:40,839 DEBUG curator.cli run:128 action_disabled = False
2020-03-25 12:56:40,839 DEBUG curator.cli run:132 continue_if_exception = False
2020-03-25 12:56:40,839 DEBUG curator.cli run:134 timeout_override = 180
2020-03-25 12:56:40,839 DEBUG curator.cli run:136 ignore_empty_list = True
2020-03-25 12:56:40,839 DEBUG curator.cli run:138 allow_ilm_indices = False
2020-03-25 12:56:40,839 INFO curator.cli run:148 Preparing Action ID: 1, "close"
2020-03-25 12:56:40,839 INFO curator.cli run:162 Creating client object and testing connection
2020-03-25 12:56:40,839 DEBUG curator.utils get_client:809 kwargs = {'hosts': ['127.0.0.1'], 'port': 9200, 'use_ssl': False, 'ssl_no_validate': False, 'master_only': False, 'url_prefix': '', 'aws_token': None, 'aws_key': None, 'http_auth': None, 'client_key': None, 'client_cert': None, 'aws_secret_key': None, 'certificate': None, 'aws_sign_request': False, 'timeout': 180}
2020-03-25 12:56:40,840 DEBUG curator.utils get_client:871 Checking for AWS settings
2020-03-25 12:56:40,844 DEBUG curator.utils get_client:886 "requests_aws4auth" module present, but not used.
2020-03-25 12:56:40,844 INFO curator.utils get_client:903 Instantiating client object
2020-03-25 12:56:40,844 INFO curator.utils get_client:906 Testing client connectivity
2020-03-25 12:56:40,848 ERROR curator.utils get_client:915 HTTP 401 error: ^M
401 Authorization Required^M
^M
401 Authorization Required
^M
nginx/1.17.5^M
^M
^M
...
So the error I'm getting is Authorization Required. How do I provide authentication? I've tried a few things with no luck. Do I need to provide certs? Which certs? The elasticsearch certs from /etc/elasticsearch/config/certs?
Any help is very much appreciated.