# Curator is failing during authentication for elastic search on AWS

**URL:** https://discuss.elastic.co/t/curator-is-failing-during-authentication-for-elastic-search-on-aws/289978
**Category:** Elasticsearch
**Created:** [November 23, 2021, 6:26pm UTC](https://discuss.elastic.co/t/curator-is-failing-during-authentication-for-elastic-search-on-aws/289978 "2021-11-23T18:26:25Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![hudar01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hudar01/32/97460_2.png) [@hudar01](https://discuss.elastic.co/u/hudar01)
#### Post date: [November 23, 2021, 6:26pm UTC](https://discuss.elastic.co/t/curator-is-failing-during-authentication-for-elastic-search-on-aws/289978/1 "2021-11-23T18:26:25Z")

</div>

I've setup an Elasticsearch backup and restore on S3, and I was trying to delete the backups using Curator (but it's always failing - i tried both ways using url\_prefix and without it). Could you please help how I can make this work? I installed and setup curator on the jumpbox, here's the curator.yaml:

```auto
--

client:

  hosts:

    - vpc-test.us-east-1.es.amazonaws.com

  url_prefix: https://vpc-test.us-east-1.es.amazonaws.com
  port: 443
  use_ssl: True
  aws_region: us-east-1
  aws_sign_request: True

  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:

  loglevel: DEBUG

  logfile: /home/ec2-user/elastic_backup/cur.log

  logformat: default

  blacklist: ['elasticsearch', 'urllib3']

```

here's the log for the curator output (cur.log):

```auto
2021-11-22 16:42:01,829 WARNING curator.utils process_auth_args:796 Use of "http_auth" is deprecated. Please use "username" and "password" instead.
2021-11-22 16:42:01,829 DEBUG curator.utils get_client:1022 kwargs = {'hosts': ['vpc-test.us-east-1.es.amazonaws.com'], 'port': 443, 'use_ssl': True, 'aws_region': 'us-east-1', 'aws_sign_request': True, 'ssl_no_validate': False, 'aws_token': 'IQoJb3JpZ==', 'client_cert': None, 'certificate': None, 'http_auth': <requests_aws4auth.aws4auth.AWS4Auth object at 0x7ff56546f0b8>, 'aws_key': 'ZCJDG2BVXU3L', 'url_prefix': '', 'client_key': None, 'aws_secret_key': 'WQHm1iBkw6WXLZ0B4VvC5p', 'timeout': 300, 'connection_class': <class 'elasticsearch.connection.http_requests.RequestsHttpConnection'>, 'verify_certs': True, 'ca_certs': '/home/ec2-user/.local/lib/python3.6/site-packages/certifi/cacert.pem'}2021-11-22 16:42:01,830 INFO curator.utils get_client:1027 Instantiating client object2021-11-22 16:42:01,861 INFO curator.utils get_client:1030 Testing client connectivity
2021-11-22 16:42:01,989 ERROR curator.utils get_client:1043 Unable to connect to Elasticsearch cluster. Error: The client noticed that the server is not Elasticsearch and we do not support this unknown product2021-11-22 16:42:01,989 CRITICAL curator.utils get_client:1048 Curator cannot proceed. Exiting.2021-11-22 18:53:40,121 ERROR Unable to connect to Elasticsearch cluster. Error: The client noticed that the server is not Elasticsearch and we do not support this unknown product
2021-11-22 18:53:40,121 CRITICAL Curator cannot proceed. Exiting.

```

hitting on line 1067 (without the url\_prefix): [curator/utils.py at 7f9461b6692e66341860600470a0ad21fcdbcb71 · elastic/curator · GitHub](https://github.com/elastic/curator/blob/7f9461b6692e66341860600470a0ad21fcdbcb71/curator/utils.py)

```auto

    LOGGER.debug("kwargs = {0}".format(kwargs))
    fail = False

    try:
        # Creating the class object should be okay
        LOGGER.info('Instantiating client object')
        client = elasticsearch.Elasticsearch(**kwargs)
        # Test client connectivity (debug log client.info() output)
        LOGGER.info('Testing client connectivity')
        LOGGER.debug('Cluster info: {0}'.format(client.info()))
        LOGGER.info('Successfully created Elasticsearch client object with provided settings')
    # Catch all TransportError types first
    except elasticsearch.TransportError as err:
        try:
            reason = err.info['error']['reason']
        except:
            reason = err.error
        LOGGER.error('HTTP {0} error: {1}'.format(err.status_code, reason))
        fail = True
    # Catch other potential exceptions
    except Exception as err:
        LOGGER.error('Unable to connect to Elasticsearch cluster. Error: {0}'.format(err))
        fail = True
    ## failure checks
    # First level failure check
    if fail:
        LOGGER.fatal('Curator cannot proceed. Exiting.')
        raise exceptions.ClientException
    # Second level failure check: acceptable version
    do_version_check(client, skip_version_test)
    # Third level failure check: master_only
    verify_master_status(client, master_only)
    return client
 

```

```auto

 With the URL_Prefix, hitting this:

2021-11-22 19:02:43,530 INFO curator.utils get_client:1027 Instantiating client object2021-11-22 19:02:43,531 INFO curator.utils get_client:1030 Testing client connectivity2021-11-22 19:02:43,557 ERROR curator.utils get_client:1039 HTTP 400 error: Validation Failed: 1: type is missing;2021-11-22 19:02:43,557 CRITICAL curator.utils get_client:1048 Curator cannot proceed. Exiting.

```

Also, going to the VPC link works, [https://vpc-test.us-east-1.es.amazonaws.com](https://vpc-test.us-east-1.es.amazonaws.com)  
this is what I get back:

```auto
{
  "name" : "<OMITTED>",
  "cluster_name" : "<OMITTED>",
  "cluster_uuid" : "<OMITTED>",
  "version" : {
    "number" : "5.6.17",
    "build_hash" : "59bb0dc",
    "build_date" : "2020-01-03T11:28:23.851Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

```

will really appreciate your help. thank you  
Huda

---

<div class="post-metadata">

### Author: ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [November 24, 2021, 4:50am UTC](https://discuss.elastic.co/t/curator-is-failing-during-authentication-for-elastic-search-on-aws/289978/2 "2021-11-24T04:50:34Z")

</div>

> [@hudar01](#):
>
> `Error: The client noticed that the server is not Elasticsearch and we do not support this unknown product`

This is because

> [@hudar01](#):
>
> `"number" : "5.6.17"`

Effectively, your Elasticsearch version is too old for the most recent version of Curator, which no longer properly detects or works with older versions of Elasticsearch.

You should probably find an older version of Curator (5.8.2 perhaps) that fully supports Elasticsearch 5.6.

---

<div class="post-metadata">

### Author: ![hudar01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hudar01/32/97460_2.png) [@hudar01](https://discuss.elastic.co/u/hudar01)
#### Post date: [November 24, 2021, 5:27pm UTC](https://discuss.elastic.co/t/curator-is-failing-during-authentication-for-elastic-search-on-aws/289978/3 "2021-11-24T17:27:58Z")

</div>

Thank you for response! I downgraded my curator to `5.8.2` as you'd suggested and it is working now..

```auto
021-11-24 17:25:38,741 DEBUG curator.validators.SchemaCheck __init__ :27 "filter" config: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'timestring': None, 'exclude': False}
2021-11-24 17:25:38,741 DEBUG curator.snapshotlist iterate_filters:527 Parsed filter args: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'timestring': None, 'exclude': False}
2021-11-24 17:25:38,741 DEBUG curator.snapshotlist iterate_filters:536 Filter args: {'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 1, 'epoch': None, 'timestring': None, 'exclude': False}
2021-11-24 17:25:38,741 DEBUG curator.snapshotlist iterate_filters:537 Pre-instance: []
2021-11-24 17:25:38,741 DEBUG curator.snapshotlist filter_by_age:273 Starting filter_by_age
2021-11-24 17:25:38,741 DEBUG curator.snapshotlist filter_by_age:276 Point of Reference: 1637688338
2021-11-24 17:25:38,741 DEBUG curator.snapshotlist iterate_filters:539 Post-instance: []
2021-11-24 17:25:38,741 INFO curator.actions.delete_snapshots do_dry_run:1161 DRY-RUN MODE. No changes will be made.
2021-11-24 17:25:38,741 INFO curator.cli run:225 Action ID: 1, "delete_snapshots" completed.
2021-11-24 17:25:38,741 INFO curator.cli run:226 Job completed.

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 22, 2021, 5:28pm UTC](https://discuss.elastic.co/t/curator-is-failing-during-authentication-for-elastic-search-on-aws/289978/4 "2021-12-22T17:28:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.