Curator problem, ES 6.1.3 / SG


(Anthony Cleaves) #1

Hello, I have recently upgraded the cluster to use 6.1.3 rather than 5.x and unfortunately curator is no longer working. Unfortunately the only error I receive is:

elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: ConnectionError(error return without exception set) caused by: SystemError(error return without exception set)

Using debug I see:

SystemError: error return without exception set
2018-02-23 16:59:48,196 DEBUG elasticsearch log_request_fail:105 > None
2018-02-23 16:59:48,197 WARNING elasticsearch mark_dead:148 Connection <Urllib3HttpConnection: https://elk-elastic-master-3.x.com:9200> has failed for 2 times in a row, putting on 120 second timeout.
Traceback (most recent call last):
File "/usr/local/bin/curator", line 11, in
sys.exit(cli())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/curator/cli.py", line 211, in cli
run(config, action_file, dry_run)
File "/usr/local/lib/python2.7/dist-packages/curator/cli.py", line 158, in run
client = get_client(**client_args)
File "/usr/local/lib/python2.7/dist-packages/curator/utils.py", line 903, in get_client

The config I am using is as follows:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - elk-elastic-master-3.x.com
    - elk-elastic-master-1.xcom
    - elk-elastic-master-2.x.com
  port: 9200
  url_prefix:
  use_ssl: True
  certificate: '/etc/elasticsearch/elk-elastic-master-1.x.com/ca-bundle.pem'
  client_cert: '/etc/elasticsearch/elk-elastic-master-1.x.com/elastic-admin.pem'
  client_key: '/etc/elasticsearch/elk-elastic-master-1.x.com/elastic-admin.key.pem'
  ssl_no_validate: False
  http_auth: x:x
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile:
  logformat: default
#  blacklist: ['elasticsearch', 'urllib3']
  blacklist: []

Has anyone faced similiar issues?

It's worth noting I am using Search Guard, and the user being used is the main admin so it has full access to the cluster. I do not see any logs from elasticsearch when trying to run curator.

I am able to telnet to each host using port 9200, so it's not a firewall issue.

Elastalert is connecting to the nodes using the same certificate,key and client_cert so I know that's working correctly.


(Anthony Cleaves) #2

Bump


(Anthony Cleaves) #3

https://github.com/bobrik/docker-curator/pull/22 This may be relevant


(Anthony Cleaves) #4

The apt version works fine:

https://packages.elastic.co/curator/5/debian/pool/main/e/elasticsearch-curator/elasticsearch-curator_5.4.1_amd64.deb


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.