Curator & AWS ElasticSearch 6.0.0 - Timeout Issue?

(Chad Upton) #1

Issue: Curator seems to get stuck or times out with no information running against ElasticSearch 6.0.0.
Curator Version: 5.4.1

Have setup the AWS Cluster using VPC Security (accessible by anything within the VPC).

Can curl the _cluster/state/metadata endpoint from the node running curator so I pretty sure it's not an access issue.

Running curator with barebones configuration of just host and loglevel of debug.

Example config:

    action: delete_indices
    description: >-
      Delete indices older than 14 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
      ignore_empty_list: True
      continue_if_exception: True
      disable_action: False
    - filtertype: pattern
      kind: prefix
      value: logstash-staging-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1


curator /opt/logstash-configs/scripts/logstash.yml
2018-01-15 19:03:41,146 DEBUG                curator.cli                    run:108  Client and logging options validated.
2018-01-15 19:03:41,146 DEBUG                curator.cli                    run:112  default_timeout = 30
2018-01-15 19:03:41,146 DEBUG                curator.cli                    run:116  action_file: /opt/logstash-configs/scripts/logstash.yml

deleted stuff here to save character limit.

(ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'continue_if_exception': True, 'disable_action': False, 'timeout_override': None}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-staging-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 1, 'stats_result': 'min_value', 'exclude': False, 'epoch': None}]}}
2018-01-15 19:03:41,154 DEBUG                curator.cli                    run:126  action_disabled = False
2018-01-15 19:03:41,154 DEBUG                curator.cli                    run:130  continue_if_exception = True
2018-01-15 19:03:41,154 DEBUG                curator.cli                    run:132  timeout_override = None
2018-01-15 19:03:41,154 DEBUG                curator.cli                    run:134  ignore_empty_list = True
2018-01-15 19:03:41,154 INFO                 curator.cli                    run:144  Preparing Action ID: 1, "delete_indices"
2018-01-15 19:03:41,155 DEBUG              curator.utils             get_client:803  kwargs = {'hosts': [''], 'aws_secret_key': None, 'master_only': False, 'port': 9200, 'certificate': None, 'aws_key': None, 'http_auth': None, 'aws_token': None, 'use_ssl': False, 'url_prefix': '', 'aws_sign_request': False, 'ssl_no_validate': False, 'client_key': None, 'client_cert': None, 'timeout': 30}
2018-01-15 19:03:41,157 DEBUG              curator.utils             get_client:880  Not using "requests_aws4auth" python module to connect.

Then just sits there. Eventually returning the prompt with no information.

(Aaron Mildenstein) #2

What do you get if you run:

curl -H'Content-Type: application/json' http://my_aws_es_host:9200

It should be something like:

  "name" : "my_aws_es_host",
  "cluster_name" : "MY_CLUSTER_NAME",
  "cluster_uuid" : "7OfiDNTsTJqiHTK4_ghIsA",
  "version" : {
    "number" : "6.1.1",
    "build_hash" : "bd92e7f",
    "build_date" : "2017-12-17T20:23:25.338Z",
    "build_snapshot" : false,
    "lucene_version" : "7.1.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  "tagline" : "You Know, for Search"

...but probably with different version numbers.

If you do not see this, or something nearly identical to it, then chances are good that Curator simply cannot connect either.

(Chad Upton) #3

I actually did get that but had a configuration error where I didn't set the port to 80 in the ~/.curator/config.yaml.

Thanks Aaron!

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.