Hello, I have recently upgraded the cluster to use 6.1.3 rather than 5.x and unfortunately curator is no longer working. Unfortunately the only error I receive is:
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch. Error: ConnectionError(error return without exception set) caused by: SystemError(error return without exception set)
Using debug I see:
SystemError: error return without exception set
2018-02-23 16:59:48,196 DEBUG elasticsearch log_request_fail:105 > None
2018-02-23 16:59:48,197 WARNING elasticsearch mark_dead:148 Connection <Urllib3HttpConnection: https://elk-elastic-master-3.x.com:9200> has failed for 2 times in a row, putting on 120 second timeout.
Traceback (most recent call last):
File "/usr/local/bin/curator", line 11, in
sys.exit(cli())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/curator/cli.py", line 211, in cli
run(config, action_file, dry_run)
File "/usr/local/lib/python2.7/dist-packages/curator/cli.py", line 158, in run
client = get_client(**client_args)
File "/usr/local/lib/python2.7/dist-packages/curator/utils.py", line 903, in get_client
The config I am using is as follows:
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- elk-elastic-master-3.x.com
- elk-elastic-master-1.xcom
- elk-elastic-master-2.x.com
port: 9200
url_prefix:
use_ssl: True
certificate: '/etc/elasticsearch/elk-elastic-master-1.x.com/ca-bundle.pem'
client_cert: '/etc/elasticsearch/elk-elastic-master-1.x.com/elastic-admin.pem'
client_key: '/etc/elasticsearch/elk-elastic-master-1.x.com/elastic-admin.key.pem'
ssl_no_validate: False
http_auth: x:x
timeout: 30
master_only: False
logging:
loglevel: DEBUG
logfile:
logformat: default
# blacklist: ['elasticsearch', 'urllib3']
blacklist: []
Has anyone faced similiar issues?
It's worth noting I am using Search Guard, and the user being used is the main admin so it has full access to the cluster. I do not see any logs from elasticsearch when trying to run curator.
I am able to telnet to each host using port 9200, so it's not a firewall issue.
Elastalert is connecting to the nodes using the same certificate,key and client_cert so I know that's working correctly.