I'm trying to register an S3 repository for a test cluster of two instances running Elasticsearch 8.12.0 in a private AWS subnet using IAM instance profiles instead of access keys. The subnet security group has outgoing internet access, and the aws s3 ls <bucket_name> command works from the host. The following command fails:
PUT _snapshot/s3-prod-elasticsearch-snapshots
{
"type": "s3",
"settings": {
"bucket": "company-name-prod-elasticsearch-snapshot-8aobgr3w"
}
}
Response:
{
"error": {
"root_cause": [
{
"type": "repository_verification_exception",
"reason": "[s3-prod-elasticsearch-snapshots] path is not accessible on master node"
}
],
"type": "repository_verification_exception",
"reason": "[s3-prod-elasticsearch-snapshots] path is not accessible on master node",
"caused_by": {
"type": "i_o_exception",
"reason": "Unable to upload object [tests-e27dT1gwQJuSkXKQp-pr5g/master.dat] using a single upload",
"caused_by": {
"type": "amazon_service_exception",
"reason": "Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)"
}
}
},
"status": 500
}
The instance profile role has the following policy:
{
"Statement": [
{
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::company-name-prod-elasticsearch-snapshot-pbvc8t7w/*",
"arn:aws:s3:::company-name-prod-elasticsearch-snapshot-pbvc8t7w"
]
}
],
"Version": "2012-10-17"
}
I thought it might be a networking issue and also tried creating an S3 gateway endpoint and associated routes, and again successfully verified it works with aws s3 ls --endpoint-url https://s3.eu-west-1.amazonaws.com company-name-prod-elasticsearch-snapshot-pbvc8t7w. But the following two commands both fail with the same error as above:
PUT _snapshot/s3-prod-elasticsearch-snapshots
{
"type": "s3",
"settings": {
"bucket": "company-name-prod-elasticsearch-snapshot-8aobgr3w",
"endpoint": "s3.eu-west-1.amazonaws.com"
}
}
PUT _snapshot/s3-prod-elasticsearch-snapshots
{
"type": "s3",
"settings": {
"bucket": "company-name-prod-elasticsearch-snapshot-8aobgr3w",
"endpoint": "s3.eu-west-1.amazonaws.com",
"server_side_encryption": "true"
}
}
elasticsearch.yml seems to contain nothing of interest:
cluster.name: 'prod-cluster'
node.name: 'elasticsearch-1'
network.bind_host: 0.0.0.0
network.publish_host: 10.0.1.50
http.port: 9200
transport.port: 9300
s3.client.default.endpoint: s3.eu-west-1.amazonaws.com
discovery.seed_hosts: '10.0.1.223'
cluster.initial_master_nodes:
- 10.0.1.50
- 10.0.1.223
xpack.license.self_generated.type: basic
xpack.monitoring.collection.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/certificates/ca/ca.crt
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certificates/elasticsearch-1/elasticsearch-1.crt
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certificates/elasticsearch-1/elasticsearch-1.key
What am I missing here, how can I get more debug information?