Hello, I have recently upgraded the cluster to use 6.1.3 rather than 5.x and unfortunately curator is no longer working. Unfortunately the only error I receive is:
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch. Error: ConnectionError(error return without exception set) caused by: SystemError(error return without exception set)
Using debug I see:
SystemError: error return without exception set
2018-02-23 16:59:48,196 DEBUG elasticsearch log_request_fail:105 > None
2018-02-23 16:59:48,197 WARNING elasticsearch mark_dead:148 Connection <Urllib3HttpConnection: https://elk-elastic-master-3.x.com:9200> has failed for 2 times in a row, putting on 120 second timeout.
Traceback (most recent call last):
File "/usr/local/bin/curator", line 11, in
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/curator/cli.py", line 211, in cli
run(config, action_file, dry_run)
File "/usr/local/lib/python2.7/dist-packages/curator/cli.py", line 158, in run
client = get_client(**client_args)
File "/usr/local/lib/python2.7/dist-packages/curator/utils.py", line 903, in get_client
The config I am using is as follows:
--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" client: hosts: - elk-elastic-master-3.x.com - elk-elastic-master-1.xcom - elk-elastic-master-2.x.com port: 9200 url_prefix: use_ssl: True certificate: '/etc/elasticsearch/elk-elastic-master-1.x.com/ca-bundle.pem' client_cert: '/etc/elasticsearch/elk-elastic-master-1.x.com/elastic-admin.pem' client_key: '/etc/elasticsearch/elk-elastic-master-1.x.com/elastic-admin.key.pem' ssl_no_validate: False http_auth: x:x timeout: 30 master_only: False logging: loglevel: DEBUG logfile: logformat: default # blacklist: ['elasticsearch', 'urllib3'] blacklist: 
Has anyone faced similiar issues?
It's worth noting I am using Search Guard, and the user being used is the main admin so it has full access to the cluster. I do not see any logs from elasticsearch when trying to run curator.
I am able to telnet to each host using port 9200, so it's not a firewall issue.
Elastalert is connecting to the nodes using the same certificate,key and client_cert so I know that's working correctly.