I'm facing a strange issue.
It was working 6 months ago.
Elasticsearch version 6.8.12 was created based on the CentOS Linux release 7.9.2009 (Core) from RPM (we are using Puppet)
{
"name": "az-es68-master-1",
"cluster_name": "az-XXXXXX-v68",
"cluster_uuid": "1suO-i2ESGGT5iSmAbaYVQ",
"version": {
"number": "6.8.12",
"build_flavor": "default",
"build_type": "rpm",
"build_hash": "7a15d2a",
"build_date": "2020-08-12T07:27:20.804867Z",
"build_snapshot": false,
"lucene_version": "7.7.3",
"minimum_wire_compatibility_version": "5.6.0",
"minimum_index_compatibility_version": "5.0.0"
},
"tagline": "You Know, for Search"
}
Installed 3 plugins
/usr/share/elasticsearch/bin/elasticsearch-plugin list
mapper-size
repository-azure
repository-s3
Credentials stored in keystore
/usr/share/elasticsearch/bin/elasticsearch-keystore list
azure.client.default.account
azure.client.default.key
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key
When I try to create a repository I see:
curl -XPUT -H 'Content-Type: application/json' 'http://localhost:9200/_snapshot/es68-prod' -d '{"type": "s3","settings": {"bucket": "XXXXX-es68","region": "us-east-2", "readonly": true}}'
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[es68-prod] path is not accessible on master node"}],"type":"repository_verification_exception","reason":"[es68-prod] path is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Exception when listing blobs by prefix [index-]","caused_by":{"type":"amazon_s3_exception","reason":"The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: DYSV72X98F8PWXG5; S3 Extended Request ID: Ywu948NpnnHtYgytOn1zswxyVXhzF+FP07DOnqbnnUOLqKVDLyvcjxR0nq5g2HMnZLso+R+1YqA=)"}}},"status":500}
What it can be?
Elasticsearch.yml is
### DO NOT MODIFY THIS FILE...
### It is managed by puppet...
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# --------------------------------- Variable ----------------------------------
cluster.name: az-XXXXXX-v68
node.name: az-es68-master-1
node.data: false
node.master: true
node.attr.tag: aws
node.attr.search: no
node.attr.indexing: no
network.host: [10.10.2.12, localhost]
path.data: /var/lib/elasticsearch/data/
path.logs: /var/lib/elasticsearch/logs/
path.repo: ["/var/backup/elasticsearch"]
http.compression: true
# --------------------------------- Discovery ---------------------------------
discovery.zen.ping.unicast.hosts:
- az-es68-master-1
- az-es68-master-2
- az-es68-master-3
discovery.zen.minimum_master_nodes: 2
# --------------------------------- X-Pack ---------------------------------
xpack.security.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: false
xpack.watcher.enabled: false
reindex.remote.whitelist: "*.*"
AWS credentials are working (double-checked and verified).
What I tried (but not helped):
- Use latest JDK (java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64)
- Use downgraded JDK (java-1.8.0-openjdk-headless-1.8.0.262.b10-0.el7_8.x86_64)
- Use the latest Elasticsearch (v.6.8.23)
- Use the latest Elasticsearch (v.7.17.1)
It was worked one year ago, and it was worked six months ago.
What are the settings I lost?
Thanks for any ideas
)