Curator is failing during authentication for elastic search on AWS

I've setup an Elasticsearch backup and restore on S3, and I was trying to delete the backups using Curator (but it's always failing - i tried both ways using url_prefix and without it). Could you please help how I can make this work? I installed and setup curator on the jumpbox, here's the curator.yaml:

--

client:

  hosts:

    - vpc-test.us-east-1.es.amazonaws.com

  url_prefix: https://vpc-test.us-east-1.es.amazonaws.com
  port: 443
  use_ssl: True
  aws_region: us-east-1
  aws_sign_request: True

  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:

  loglevel: DEBUG

  logfile: /home/ec2-user/elastic_backup/cur.log

  logformat: default

  blacklist: ['elasticsearch', 'urllib3']

here's the log for the curator output (cur.log):

2021-11-22 16:42:01,829 WARNING            curator.utils      process_auth_args:796  Use of "http_auth" is deprecated. Please use "username" and "password" instead.
2021-11-22 16:42:01,829 DEBUG              curator.utils             get_client:1022 kwargs = {'hosts': ['vpc-test.us-east-1.es.amazonaws.com'], 'port': 443, 'use_ssl': True, 'aws_region': 'us-east-1', 'aws_sign_request': True, 'ssl_no_validate': False, 'aws_token': 'IQoJb3JpZ==', 'client_cert': None, 'certificate': None, 'http_auth': <requests_aws4auth.aws4auth.AWS4Auth object at 0x7ff56546f0b8>, 'aws_key': 'ZCJDG2BVXU3L', 'url_prefix': '', 'client_key': None, 'aws_secret_key': 'WQHm1iBkw6WXLZ0B4VvC5p', 'timeout': 300, 'connection_class': <class 'elasticsearch.connection.http_requests.RequestsHttpConnection'>, 'verify_certs': True, 'ca_certs': '/home/ec2-user/.local/lib/python3.6/site-packages/certifi/cacert.pem'}2021-11-22 16:42:01,830 INFO               curator.utils             get_client:1027 Instantiating client object2021-11-22 16:42:01,861 INFO               curator.utils             get_client:1030 Testing client connectivity
2021-11-22 16:42:01,989 ERROR              curator.utils             get_client:1043 Unable to connect to Elasticsearch cluster. Error: The client noticed that the server is not Elasticsearch and we do not support this unknown product2021-11-22 16:42:01,989 CRITICAL           curator.utils             get_client:1048 Curator cannot proceed. Exiting.2021-11-22 18:53:40,121 ERROR     Unable to connect to Elasticsearch cluster. Error: The client noticed that the server is not Elasticsearch and we do not support this unknown product
2021-11-22 18:53:40,121 CRITICAL  Curator cannot proceed. Exiting.

hitting on line 1067 (without the url_prefix): curator/utils.py at 7f9461b6692e66341860600470a0ad21fcdbcb71 · elastic/curator · GitHub


    LOGGER.debug("kwargs = {0}".format(kwargs))
    fail = False

    try:
        # Creating the class object should be okay
        LOGGER.info('Instantiating client object')
        client = elasticsearch.Elasticsearch(**kwargs)
        # Test client connectivity (debug log client.info() output)
        LOGGER.info('Testing client connectivity')
        LOGGER.debug('Cluster info: {0}'.format(client.info()))
        LOGGER.info('Successfully created Elasticsearch client object with provided settings')
    # Catch all TransportError types first
    except elasticsearch.TransportError as err:
        try:
            reason = err.info['error']['reason']
        except:
            reason = err.error
        LOGGER.error('HTTP {0} error: {1}'.format(err.status_code, reason))
        fail = True
    # Catch other potential exceptions
    except Exception as err:
        LOGGER.error('Unable to connect to Elasticsearch cluster. Error: {0}'.format(err))
        fail = True
    ## failure checks
    # First level failure check
    if fail:
        LOGGER.fatal('Curator cannot proceed. Exiting.')
        raise exceptions.ClientException
    # Second level failure check: acceptable version
    do_version_check(client, skip_version_test)
    # Third level failure check: master_only
    verify_master_status(client, master_only)
    return client
 

 With the URL_Prefix, hitting this:

2021-11-22 19:02:43,530 INFO               curator.utils             get_client:1027 Instantiating client object2021-11-22 19:02:43,531 INFO               curator.utils             get_client:1030 Testing client connectivity2021-11-22 19:02:43,557 ERROR              curator.utils             get_client:1039 HTTP 400 error: Validation Failed: 1: type is missing;2021-11-22 19:02:43,557 CRITICAL           curator.utils             get_client:1048 Curator cannot proceed. Exiting.

Also, going to the VPC link works, https://vpc-test.us-east-1.es.amazonaws.com
this is what I get back:

{
  "name" : "<OMITTED>",
  "cluster_name" : "<OMITTED>",
  "cluster_uuid" : "<OMITTED>",
  "version" : {
    "number" : "5.6.17",
    "build_hash" : "59bb0dc",
    "build_date" : "2020-01-03T11:28:23.851Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

will really appreciate your help. thank you
Huda

This is because

Effectively, your Elasticsearch version is too old for the most recent version of Curator, which no longer properly detects or works with older versions of Elasticsearch.

You should probably find an older version of Curator (5.8.2 perhaps) that fully supports Elasticsearch 5.6.

Thank you for response! I downgraded my curator to 5.8.2 as you'd suggested and it is working now..

021-11-24 17:25:38,741 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'timestring': None, 'exclude': False}
2021-11-24 17:25:38,741 DEBUG       curator.snapshotlist        iterate_filters:527  Parsed filter args: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'timestring': None, 'exclude': False}
2021-11-24 17:25:38,741 DEBUG       curator.snapshotlist        iterate_filters:536  Filter args: {'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'timestring': None, 'exclude': False}
2021-11-24 17:25:38,741 DEBUG       curator.snapshotlist        iterate_filters:537  Pre-instance: []
2021-11-24 17:25:38,741 DEBUG       curator.snapshotlist          filter_by_age:273  Starting filter_by_age
2021-11-24 17:25:38,741 DEBUG       curator.snapshotlist          filter_by_age:276  Point of Reference: 1637688338
2021-11-24 17:25:38,741 DEBUG       curator.snapshotlist        iterate_filters:539  Post-instance: []
2021-11-24 17:25:38,741 INFO      curator.actions.delete_snapshots             do_dry_run:1161 DRY-RUN MODE.  No changes will be made.
2021-11-24 17:25:38,741 INFO                 curator.cli                    run:225  Action ID: 1, "delete_snapshots" completed.
2021-11-24 17:25:38,741 INFO                 curator.cli                    run:226  Job completed.