Curator & AWS ElasticSearch 6.0.0 - Timeout Issue?

[Hush077 (Chad Upton)]

Issue: Curator seems to get stuck or times out with no information running against ElasticSearch 6.0.0.
Curator Version: 5.4.1

Have setup the AWS Cluster using VPC Security (accessible by anything within the VPC).

Can curl the _cluster/state/metadata endpoint from the node running curator so I pretty sure it's not an access issue.

Running curator with barebones configuration of just host and loglevel of debug.

Example config:

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 14 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      continue_if_exception: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-staging-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1

Logfile:

curator /opt/logstash-configs/scripts/logstash.yml
2018-01-15 19:03:41,146 DEBUG                curator.cli                    run:108  Client and logging options validated.
2018-01-15 19:03:41,146 DEBUG                curator.cli                    run:112  default_timeout = 30
2018-01-15 19:03:41,146 DEBUG                curator.cli                    run:116  action_file: /opt/logstash-configs/scripts/logstash.yml

deleted stuff here to save character limit.

(ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'continue_if_exception': True, 'disable_action': False, 'timeout_override': None}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-staging-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}]}}
2018-01-15 19:03:41,154 DEBUG                curator.cli                    run:126  action_disabled = False
2018-01-15 19:03:41,154 DEBUG                curator.cli                    run:130  continue_if_exception = True
2018-01-15 19:03:41,154 DEBUG                curator.cli                    run:132  timeout_override = None
2018-01-15 19:03:41,154 DEBUG                curator.cli                    run:134  ignore_empty_list = True
2018-01-15 19:03:41,154 INFO                 curator.cli                    run:144  Preparing Action ID: 1, "delete_indices"
2018-01-15 19:03:41,155 DEBUG              curator.utils             get_client:803  kwargs = {'hosts': ['vpc-logs-somelongid.us-east-1.es.amazonaws.com'], 'aws_secret_key': None, 'master_only': False, 'port': 9200, 'certificate': None, 'aws_key': None, 'http_auth': None, 'aws_token': None, 'use_ssl': False, 'url_prefix': '', 'aws_sign_request': False, 'ssl_no_validate': False, 'client_key': None, 'client_cert': None, 'timeout': 30}
2018-01-15 19:03:41,157 DEBUG              curator.utils             get_client:880  Not using "requests_aws4auth" python module to connect.

Then just sits there. Eventually returning the prompt with no information.

[theuntergeek (Aaron Mildenstein)]

What do you get if you run:

curl -H'Content-Type: application/json' http://my_aws_es_host:9200

It should be something like:

{
  "name" : "my_aws_es_host",
  "cluster_name" : "MY_CLUSTER_NAME",
  "cluster_uuid" : "7OfiDNTsTJqiHTK4_ghIsA",
  "version" : {
    "number" : "6.1.1",
    "build_hash" : "bd92e7f",
    "build_date" : "2017-12-17T20:23:25.338Z",
    "build_snapshot" : false,
    "lucene_version" : "7.1.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

...but probably with different version numbers.

If you do not see this, or something nearly identical to it, then chances are good that Curator simply cannot connect either.

[Hush077 (Chad Upton)]

I actually did get that but had a configuration error where I didn't set the port to 80 in the ~/.curator/config.yaml.

Thanks Aaron!