Curator run failing with NewConnectionError(urllib3.connection.HTTPConnection)

Hello,

When I am trying to delete index on ELK single node cluster using curator it is failing with below error. I know it is a common error having many threads, but I am not able to find a proper solution for any.

When I try to delete using curator, it is failing with 'failed to establish connection error(Errno[-2]). Complete error snippets are given below.

Software versions used

elasticsearch-curator (5.7.6)
urllib3 (1.24.3)
elasticsearch = 6.2.4

Additional tests done to make sure ELK is working fine!

  1. 9200 port is up and running on ELK
  2. Curl GET request on the host:port is returning success json
  3. Python client using urllib3 is able to connect elk port 9200

Error prints observed during script run:-


 [ec2-user@ip-172-31-26-150 curator]$ curator curator-action.yml --config curator-config.yml 
2019-07-16 13:37:13,061 INFO      Preparing Action ID: 1, "delete_indices"
Traceback (most recent call last):
  File "/usr/local/bin/curator", line 11, in <module>
    load_entry_point('elasticsearch-curator==5.7.6', 'console_scripts', 'curator')()
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/curator/cli.py", line 218, in cli
    run(config, action_file, dry_run)
  File "/usr/local/lib/python2.7/site-packages/curator/cli.py", line 165, in run
    client = get_client(**client_args)
  File "/usr/local/lib/python2.7/site-packages/curator/utils.py", line 911, in get_client
    'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe4e2149bd0>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe4e2149bd0>: Failed to establish a new connection: [Errno -2] Name or service not known)

Additional verification done on ELK :-

Python urllib3 client to GET ELK info

>>> import urllib3
>>> http = urllib3.PoolManager()
>>> r = http.request('GET', 'http://172.31.29.21:9200')
>>> r.status
200

CURL request to GET ELK info

[ec2-user@ip-172-31-26-150 curator]$ curl -X GET http://172.31.29.21:9200
{
  "name" : "EaR2f-Q",
  "cluster_name" : "elk-single-pagerduty",
  "cluster_uuid" : "pfRgoQ-JQ-WkkFvNVNwn3g",
  "version" : {
    "number" : "6.2.4",
    "build_hash" : "ccec39f",
    "build_date" : "2018-04-12T20:37:28.497551Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Also attaching curator config and action file

curator-config.yml file output

client:
  hosts:
    — 172.31.29.21
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  aws_key:
  aws_secret_key:
  aws_region:
  ssl_no_validate: False
  http_auth:
  timeout: 100
  master_only: False
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

curator-action.yml file output

--- 
actions: 
  1: 
    action: delete_indices
    description: "Delete indices older than 30 days (based on index name)."
    filters: 
      - 
        exclude: ~
        filtertype: pattern
        kind: prefix
        value: logstash-
      - 
        direction: older
        exclude: ~
        filtertype: age
        source: name
        timestring: ‘%Y.%m.%d’
        unit: days
        unit_count: 60
    options: 
      continue_if_exception: false
      disable_action: false
      ignore_empty_list: true
      timeout_override: ~

Any help is really appreciated!

Thanks in advance.
Sanoop

Please reformat your pasted data. It is simply too difficult to read in this format. All pasted data should be between two rows of triple back ticks, like this:

```
PASTE HERE
```

Please edit your original post to reformat the data. I will look after that.

Updated. Can you please look into the issue?

You can test the same way Curator does, using the elasticsearch python module itself:

$ python
Python 3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import elasticsearch
>>> c = elasticsearch.Elasticsearch()
>>> c.info()
{'name': 'myhost.local', 'cluster_name': 'foo', 'cluster_uuid': 'wGyxgqg8RU6n8Mq_d27AAQ', 'version': {'number': '7.2.0', 'build_flavor': 'default', 'build_type': 'tar', 'build_hash': '508c38a', 'build_date': '2019-06-20T15:54:18.811730Z', 'build_snapshot': False, 'lucene_version': '8.0.0', 'minimum_wire_compatibility_version': '6.8.0', 'minimum_index_compatibility_version': '6.0.0-beta1'}, 'tagline': 'You Know, for Search'}

This uses the defaults for everything, so http://localhost:9200, but you can set hosts=["a.b.c.d"], port=9200 to specify manually.

You can also test the exact function Curator uses, if you install Curator via pip:

$ python
Python 3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import curator
>>> c = curator.utils.get_client(hosts=["localhost"], port=9200)
>>> c.info()
{'name': 'myhost.local', 'cluster_name': 'foo', 'cluster_uuid': 'wGyxgqg8RU6n8Mq_d27AAQ', 'version': {'number': '7.2.0', 'build_flavor': 'default', 'build_type': 'tar', 'build_hash': '508c38a', 'build_date': '2019-06-20T15:54:18.811730Z', 'build_snapshot': False, 'lucene_version': '8.0.0', 'minimum_wire_compatibility_version': '6.8.0', 'minimum_index_compatibility_version': '6.0.0-beta1'}, 'tagline': 'You Know, for Search'}

You can also see what urllib3 is doing more directly by setting blacklist: [] (an empty list). This will show everything the elasticsearch python module attempts, as well as what it passes through urllib3.

Thank you!

It is working fine with elasticsearch module and curator module if I try this via python command line.

[ec2-user@ip-172-31-26-150 curator]$ python
Python 2.7.16 (default, Mar 18 2019, 18:38:44) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import curator
>>> c = curator.utils.get_client(hosts=["172.31.29.21"], port=9200)
>>> c.info()
{u'cluster_name': u'elk-single-pagerduty', u'cluster_uuid': u'pfRgoQ-JQ-WkkFvNVNwn3g', u'version': {u'build_date': u'2018-04-12T20:37:28.497551Z', u'minimum_wire_compatibility_version': u'5.6.0', u'build_hash': u'ccec39f', u'number': u'6.2.4', u'minimum_index_compatibility_version': u'5.0.0', u'build_snapshot': False, u'lucene_version': u'7.2.1'}, u'name': u'EaR2f-Q', u'tagline': u'You Know, for Search'}
>>> 
[ec2-user@ip-172-31-26-150 curator]$ python
Python 2.7.16 (default, Mar 18 2019, 18:38:44) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import elasticsearch
>>> c = elasticsearch.Elasticsearch(hosts=["172.31.29.21"], port=9200)
>>> c.info()
{u'cluster_name': u'elk-single-pagerduty', u'cluster_uuid': u'pfRgoQ-JQ-WkkFvNVNwn3g', u'version': {u'build_date': u'2018-04-12T20:37:28.497551Z', u'minimum_wire_compatibility_version': u'5.6.0', u'build_hash': u'ccec39f', u'number': u'6.2.4', u'minimum_index_compatibility_version': u'5.0.0', u'build_snapshot': False, u'lucene_version': u'7.2.1'}, u'name': u'EaR2f-Q', u'tagline': u'You Know, for Search'}
>>> 

I am attaching the detailed error I am getting. It is still complaining about new connection error.

2019-07-17 09:25:38,523 WARNING   GET http://— 172.31.29.21:9200/ [status:N/A request:0.003s]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python2.7/site-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fad44425710>: Failed to establish a new connection: [Errno -2] Name or service not known


2019-07-17 09:25:45,534 WARNING   GET http://— 172.31.29.21:9200/ [status:N/A request:0.003s]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python2.7/site-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fad44425bd0>: Failed to establish a new connection: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "/usr/local/bin/curator", line 11, in <module>
    load_entry_point('elasticsearch-curator==5.7.6', 'console_scripts', 'curator')()
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/curator/cli.py", line 218, in cli
    run(config, action_file, dry_run)
  File "/usr/local/lib/python2.7/site-packages/curator/cli.py", line 165, in run
    client = get_client(**client_args)
  File "/usr/local/lib/python2.7/site-packages/curator/utils.py", line 911, in get_client
    'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fad44425bd0>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fad44425bd0>: Failed to establish a new connection: [Errno -2] Name or service not known)

This appears to be the issue. It seems to be a different hyphen, so YAML is interpreting it as part of a string. You can see that it is putting the dash inline with your host name, which is why it is unable to connect. As to how or why this happened, I can only speculate. Perhaps some text editor injected it or replaced it with a longer emdash?

You do not have to specify hosts in a multi-line manner. You can also specify the single host you have inside square braces:

hosts: [ "172.31.29.21" ]

I worry, however, that if you had this dash replaced by an emdash, that the same could be true with the dashes in your action YAML file.

Just figured out the issue. It was one special character in ELK URL in curator-config file. After removing that everything worked!

Thank you very much for the help.

Yes, fixed that hyphen and it is working fine. Copy paste of code with hyphen created the issue!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.