Using Curator

I've been tasked with setting up Curator to delete data more than 40 days old on our existing ELK 6.8.6 server. I did not set up elk and am pretty much unfamiliar with how it is configured.

I set up the yum repository, installed the latest curator. I set up a (default) config.yml and action.yml.
...
config file:
*# cat /etc/curator/config.yml *
client:

  • hosts:*
    • 127.0.0.1*
  • port: 9200*
  • url_prefix:*
  • use_ssl: False*
  • certificate:*
  • client_cert:*
  • client_key:*
  • ssl_no_validate: False*
  • http_auth: *
  • timeout: 30*
  • master_only: False*

logging:

  • loglevel: DEBUG*
  • logfile: /home/bclegg/testlog*
  • logformat: default*
  • blacklist: ['elasticsearch', 'urllib3']*
    ...
    action file:
    ...

cat /etc/curator/action.yml

actions:

  • 1:*

  • action: close*

  • description: >-*

  •  Close indices older than 40 days (based on index name).*
    
  • options:*

  •  ignore_empty_list: True*
    
  •  delete_aliases: False*
    
  •  disable_action: False*
    
  • filters:*

    • filtertype: pattern*
  •  kind: prefix*
    
  •  value: nginx-*
    
    • filtertype: age*
  •  source: name*
    
  •  direction: older*
    
  •  timestring: '%Y.%m.%d'*
    
  •  unit: days*
    
  •  unit_count: 40*
    
  • 2:*

  • action: delete_indices*

  • description: >-*

  •  Delete indices older than 40 days (based on index name).*
    
  • options:*

  •  ignore_empty_list: True*
    
  •  disable_action: False*
    
  • filters:*

    • filtertype: pattern*
  •  kind: prefix*
    
  •  value: nginx-*
    
    • filtertype: age*
  •  source: name*
    
  •  direction: older*
    
  •  timestring: '%Y.%m.%d'*
    
  •  unit: days*
    
  •  unit_count: 40*
    

...

fwiw, I chose nginx as a place to start. Once I get this to work, I'll expand.

When I attempt a dry run, I get:

/usr/bin/curator --dry-run /etc/curator/action.yml

...
2020-03-25 12:56:40,839 DEBUG curator.cli run:128 action_disabled = False
2020-03-25 12:56:40,839 DEBUG curator.cli run:132 continue_if_exception = False
2020-03-25 12:56:40,839 DEBUG curator.cli run:134 timeout_override = 180
2020-03-25 12:56:40,839 DEBUG curator.cli run:136 ignore_empty_list = True
2020-03-25 12:56:40,839 DEBUG curator.cli run:138 allow_ilm_indices = False
2020-03-25 12:56:40,839 INFO curator.cli run:148 Preparing Action ID: 1, "close"
2020-03-25 12:56:40,839 INFO curator.cli run:162 Creating client object and testing connection
2020-03-25 12:56:40,839 DEBUG curator.utils get_client:809 kwargs = {'hosts': ['127.0.0.1'], 'port': 9200, 'use_ssl': False, 'ssl_no_validate': False, 'master_only': False, 'url_prefix': '', 'aws_token': None, 'aws_key': None, 'http_auth': None, 'client_key': None, 'client_cert': None, 'aws_secret_key': None, 'certificate': None, 'aws_sign_request': False, 'timeout': 180}
2020-03-25 12:56:40,840 DEBUG curator.utils get_client:871 Checking for AWS settings
2020-03-25 12:56:40,844 DEBUG curator.utils get_client:886 "requests_aws4auth" module present, but not used.
2020-03-25 12:56:40,844 INFO curator.utils get_client:903 Instantiating client object
2020-03-25 12:56:40,844 INFO curator.utils get_client:906 Testing client connectivity
2020-03-25 12:56:40,848 ERROR curator.utils get_client:915 HTTP 401 error: ^M
401 Authorization Required^M
^M

401 Authorization Required

^M

nginx/1.17.5^M

^M
^M
...
So the error I'm getting is Authorization Required. How do I provide authentication? I've tried a few things with no luck. Do I need to provide certs? Which certs? The elasticsearch certs from /etc/elasticsearch/config/certs?

Any help is very much appreciated.

I tried pointing to the elasticsearch certs - now I'm seeing this in the log:

"/etc/curator/config.yml" 19L, 407C written
2020-03-25 13:35:09,271 DEBUG curator.cli run:128 action_disabled = False
2020-03-25 13:35:09,271 DEBUG curator.cli run:132 continue_if_exception = False
2020-03-25 13:35:09,271 DEBUG curator.cli run:134 timeout_override = 180
2020-03-25 13:35:09,271 DEBUG curator.cli run:136 ignore_empty_list = True
2020-03-25 13:35:09,271 DEBUG curator.cli run:138 allow_ilm_indices = False
2020-03-25 13:35:09,271 INFO curator.cli run:148 Preparing Action ID: 1, "close"
2020-03-25 13:35:09,271 INFO curator.cli run:162 Creating client object and testing connection
2020-03-25 13:35:09,271 DEBUG curator.utils get_client:809 kwargs = {'hosts': ['127.0.0.1'], 'port': 9200, 'use_ssl': True, 'client_cert': '/etc/elasticsearch/config/certs/elk/elk.crt', 'client_key': '/etc/elasticsearch/config/certs/elk/elk.key', 'ssl_no_validate': False, 'master_only': False, 'url_prefix': '', 'aws_token': None, 'certificate': None, 'aws_secret_key': None, 'http_auth': None, 'aws_key': None, 'aws_sign_request': False, 'timeout': 180}
2020-03-25 13:35:09,271 DEBUG curator.utils get_client:815 Attempting to verify SSL certificate.
2020-03-25 13:35:09,272 DEBUG curator.utils get_client:871 Checking for AWS settings
2020-03-25 13:35:09,276 DEBUG curator.utils get_client:886 "requests_aws4auth" module present, but not used.
2020-03-25 13:35:09,276 INFO curator.utils get_client:903 Instantiating client object
2020-03-25 13:35:09,276 INFO curator.utils get_client:906 Testing client connectivity
2020-03-25 13:35:09,403 ERROR curator.utils get_client:915 HTTP N/A error: HTTPSConnectionPool(host='127.0.0.1', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)')))
2020-03-25 13:35:09,404 CRITICAL curator.utils get_client:923 Curator cannot proceed. Exiting.

So, it appears that you were able to connect to the cluster without certs before, it merely needed a username and password.

You should define http_auth to set username and password.

Thank you for your quick reply.

I'm not sure I've ever been connected with curator. I think I have not.

I've tried several more changes to the config.yml file here is my latest attempt with some of the information screened out:

/usr/bin/curator --dry-run /etc/curator/action.yml

from the logfile:

client:
hosts:
- 10.X.X.X
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert: /etc/elasticsearch/config/certs/elk/elk.crt
client_key: /etc/elasticsearch/config/certs/elk/elk.key
ssl_no_validate: False
http_auth: root:
timeout: 30
master_only: False

logging:
loglevel: DEBUG
logfile: /home/bclegg/testlog
logformat: default
blacklist: ['elasticsearch', 'urllib3']
~

~
"/etc/curator/config.yml" 19L, 427C written
2020-03-25 15:55:39,407 DEBUG curator.cli run:110 Client and logging options validated.
2020-03-25 15:55:39,407 DEBUG curator.cli run:114 default_timeout = 30
2020-03-25 15:55:39,408 DEBUG curator.cli run:118 action_file: /etc/curator/action.yml
2020-03-25 15:55:39,420 DEBUG curator.cli run:120 action_config: {'actions': {1: {'action': 'close', 'description': 'Close indices older than 40 days (based on index name).', 'options': {'ignore_empty_list': True, 'delete_aliases': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 40}]}, 2: {'action': 'delete_indices', 'description': 'Delete indices older than 40 days (based on index name).', 'options': {'ignore_empty_list': True, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 40}]}}}
2020-03-25 15:55:39,420 DEBUG curator.validators.SchemaCheck init:26 Schema: {'actions': <class 'dict'>}
2020-03-25 15:55:39,420 DEBUG curator.validators.SchemaCheck init:27 "Actions File" config: {'actions': {1: {'action': 'close', 'description': 'Close indices older than 40 days (based on index name).', 'options': {'ignore_empty_list': True, 'delete_aliases': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 40}]}, 2: {'action': 'delete_indices', 'description': 'Delete indices older than 40 days (based on index name).', 'options': {'ignore_empty_list': True, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 40}]}}}
2020-03-25 15:55:39,421 DEBUG curator.validators.SchemaCheck init:26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'freeze', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot', 'unfreeze']")}
2020-03-25 15:55:39,421 DEBUG curator.validators.SchemaCheck init:27 "action type" config: {'action': 'close', 'description': 'Close indices older than 40 days (based on index name).', 'options': {'ignore_empty_list': True, 'delete_aliases': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'nginx-'}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 40}]}
@ @ @ @ @ @ @ @
"testlog" [readonly] 58L, 20033C
2020-03-25 15:55:39,439 DEBUG curator.cli run:128 action_disabled = False
2020-03-25 15:55:39,439 DEBUG curator.cli run:132 continue_if_exception = False
2020-03-25 15:55:39,439 DEBUG curator.cli run:134 timeout_override = 180
2020-03-25 15:55:39,439 DEBUG curator.cli run:136 ignore_empty_list = True
2020-03-25 15:55:39,439 DEBUG curator.cli run:138 allow_ilm_indices = False
2020-03-25 15:55:39,439 INFO curator.cli run:148 Preparing Action ID: 1, "close"
2020-03-25 15:55:39,440 INFO curator.cli run:162 Creating client object and testing connection
2020-03-25 15:55:39,440 DEBUG curator.utils get_client:809 kwargs = {'hosts': ['10.X.X.X'], 'port': 9200, 'use_ssl': False, 'client_cert': '/etc/elasticsearch/config/certs/elk/elk.crt', 'client_key': '/etc/elasticsearch/config/certs/elk/elk.key', 'ssl_no_validate': False, 'http_auth': 'root:', 'master_only': False, 'aws_secret_key': None, 'url_prefix': '', 'aws_key': None, 'certificate': None, 'aws_token': None, 'aws_sign_request': False, 'timeout': 180}
2020-03-25 15:55:39,440 DEBUG curator.utils get_client:871 Checking for AWS settings
2020-03-25 15:55:39,446 DEBUG curator.utils get_client:886 "requests_aws4auth" module present, but not used.
2020-03-25 15:55:39,446 INFO curator.utils get_client:903 Instantiating client object
2020-03-25 15:55:39,447 INFO curator.utils get_client:906 Testing client connectivity
2020-03-25 15:55:39,452 ERROR curator.utils get_client:915 HTTP 401 error: ^M

401 Authorization Required^M ^M

401 Authorization Required

^M
nginx/1.17.5^M ^M ^M

2020-03-25 15:55:39,452 CRITICAL curator.utils get_client:923 Curator cannot proceed. Exiting.

So, this is asking for Elasticsearch credentials, rather than operating system level credentials. Is there someone on your team who could help you get super-user credentials to Elasticsearch?

I asked and all they had was a kibana user. I tried it and I was able process action.yml without any errors in the logfile.

Thank You

1 Like