I've setup an Elasticsearch backup and restore on S3, and I was trying to delete the backups using Curator (but it's always failing - i tried both ways using url_prefix and without it). Could you please help how I can make this work? I installed and setup curator on the jumpbox, here's the curator.yaml:
--
client:
hosts:
- vpc-test.us-east-1.es.amazonaws.com
url_prefix: https://vpc-test.us-east-1.es.amazonaws.com
port: 443
use_ssl: True
aws_region: us-east-1
aws_sign_request: True
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False
logging:
loglevel: DEBUG
logfile: /home/ec2-user/elastic_backup/cur.log
logformat: default
blacklist: ['elasticsearch', 'urllib3']
here's the log for the curator output (cur.log):
2021-11-22 16:42:01,829 WARNING curator.utils process_auth_args:796 Use of "http_auth" is deprecated. Please use "username" and "password" instead.
2021-11-22 16:42:01,829 DEBUG curator.utils get_client:1022 kwargs = {'hosts': ['vpc-test.us-east-1.es.amazonaws.com'], 'port': 443, 'use_ssl': True, 'aws_region': 'us-east-1', 'aws_sign_request': True, 'ssl_no_validate': False, 'aws_token': 'IQoJb3JpZ==', 'client_cert': None, 'certificate': None, 'http_auth': <requests_aws4auth.aws4auth.AWS4Auth object at 0x7ff56546f0b8>, 'aws_key': 'ZCJDG2BVXU3L', 'url_prefix': '', 'client_key': None, 'aws_secret_key': 'WQHm1iBkw6WXLZ0B4VvC5p', 'timeout': 300, 'connection_class': <class 'elasticsearch.connection.http_requests.RequestsHttpConnection'>, 'verify_certs': True, 'ca_certs': '/home/ec2-user/.local/lib/python3.6/site-packages/certifi/cacert.pem'}2021-11-22 16:42:01,830 INFO curator.utils get_client:1027 Instantiating client object2021-11-22 16:42:01,861 INFO curator.utils get_client:1030 Testing client connectivity
2021-11-22 16:42:01,989 ERROR curator.utils get_client:1043 Unable to connect to Elasticsearch cluster. Error: The client noticed that the server is not Elasticsearch and we do not support this unknown product2021-11-22 16:42:01,989 CRITICAL curator.utils get_client:1048 Curator cannot proceed. Exiting.2021-11-22 18:53:40,121 ERROR Unable to connect to Elasticsearch cluster. Error: The client noticed that the server is not Elasticsearch and we do not support this unknown product
2021-11-22 18:53:40,121 CRITICAL Curator cannot proceed. Exiting.
hitting on line 1067 (without the url_prefix): curator/utils.py at 7f9461b6692e66341860600470a0ad21fcdbcb71 · elastic/curator · GitHub
LOGGER.debug("kwargs = {0}".format(kwargs))
fail = False
try:
# Creating the class object should be okay
LOGGER.info('Instantiating client object')
client = elasticsearch.Elasticsearch(**kwargs)
# Test client connectivity (debug log client.info() output)
LOGGER.info('Testing client connectivity')
LOGGER.debug('Cluster info: {0}'.format(client.info()))
LOGGER.info('Successfully created Elasticsearch client object with provided settings')
# Catch all TransportError types first
except elasticsearch.TransportError as err:
try:
reason = err.info['error']['reason']
except:
reason = err.error
LOGGER.error('HTTP {0} error: {1}'.format(err.status_code, reason))
fail = True
# Catch other potential exceptions
except Exception as err:
LOGGER.error('Unable to connect to Elasticsearch cluster. Error: {0}'.format(err))
fail = True
## failure checks
# First level failure check
if fail:
LOGGER.fatal('Curator cannot proceed. Exiting.')
raise exceptions.ClientException
# Second level failure check: acceptable version
do_version_check(client, skip_version_test)
# Third level failure check: master_only
verify_master_status(client, master_only)
return client
With the URL_Prefix, hitting this:
2021-11-22 19:02:43,530 INFO curator.utils get_client:1027 Instantiating client object2021-11-22 19:02:43,531 INFO curator.utils get_client:1030 Testing client connectivity2021-11-22 19:02:43,557 ERROR curator.utils get_client:1039 HTTP 400 error: Validation Failed: 1: type is missing;2021-11-22 19:02:43,557 CRITICAL curator.utils get_client:1048 Curator cannot proceed. Exiting.
Also, going to the VPC link works, https://vpc-test.us-east-1.es.amazonaws.com
this is what I get back:
{
"name" : "<OMITTED>",
"cluster_name" : "<OMITTED>",
"cluster_uuid" : "<OMITTED>",
"version" : {
"number" : "5.6.17",
"build_hash" : "59bb0dc",
"build_date" : "2020-01-03T11:28:23.851Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}
will really appreciate your help. thank you
Huda