Hi, I am working on Snapshot and Restore to ensure that my Elasticsearch indices are securely backed-up and stored. I've set up an EFK stack on my AWS EKS cluster. I've deployed the stack using Helm in Bitnami chart. I am using the recent version (8.6.2) of Elasticsearch and Kibana.
I've created an AWS S3 bucket with a name, say, my-eks-cluster-logging-snapshot-repository, and registered a repository using Kibana UI. Here are the details of the registered repository.
Type: AWS S3
Client: default
Bucket: my-eks-cluster-logging-snapshot-repository
Base path: Left as blank
Snapshot compression: Yes
Chunk size: Left as blank
Buffer size: Left as blank
Canned ACL: private
Storage class: standard
Max snapshot bytes per second: Left as blank
Max restore bytes per second: Left as blank
Read-only: No
I want to use IAM roles for Kubernetes service accounts for authentication, so I've checked the documentation about it in this link.
I've created repository-s3
directory in the Elasticsearch configuration path of elasticsearch-master nodes, and have created a symbolic link as said in the documentation.
However, when I click on Verify repository in Kibana UI, it fails and shows following error message.
{
"name": "ResponseError",
"meta": {
"body": {
"error": {
"root_cause": [
{
"type": "repository_verification_exception",
"reason": "[my-eks-cluster-logging-snapshot-repository] path is not accessible on master node"
}
],
"type": "repository_verification_exception",
"reason": "[my-eks-cluster-logging-snapshot-repository] path is not accessible on master node",
"caused_by": {
"type": "i_o_exception",
"reason": "Unable to upload object [tests-<REDACTED>/master.dat] using a single upload",
"caused_by": {
"type": "amazon_s3_exception",
"reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <REDACTED>; S3 Extended Request ID: <REDACTED>/<REDACTED>/<REDACTED>=; Proxy: null)"
}
}
},
"status": 500
},
"statusCode": 500,
"headers": {
"x-opaque-id": "bb3a5e7c-dea5-4b31-9b6c-<REDACTED>;kibana:application:management:",
"x-elastic-product": "Elasticsearch",
"content-type": "application/json;charset=utf-8",
"content-length": "764"
},
"meta": {
"context": null,
"request": {
"params": {
"method": "POST",
"path": "/_snapshot/my-eks-cluster-logging-snapshot-repository/_verify",
"querystring": "",
"headers": {
"user-agent": "Kibana/8.6.2",
"x-elastic-product-origin": "kibana",
"x-opaque-id": "bb3a5e7c-dea5-4b31-9b6c-<REDACTED>;kibana:application:management:",
"x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1",
"accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
}
},
"options": {
"opaqueId": "bb3a5e7c-dea5-4b31-9b6c-<REDACTED>;kibana:application:management:",
"headers": {
"x-elastic-product-origin": "kibana",
"user-agent": "Kibana/8.6.2",
"x-opaque-id": "bb3a5e7c-dea5-4b31-9b6c-<REDACTED>",
"x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"
}
},
"id": 1
},
"name": "elasticsearch-js",
"connection": {
"url": "http://elasticsearch:9200/",
"id": "http://elasticsearch:9200/",
"headers": {},
"status": "alive"
},
"attempts": 0,
"aborted": false
},
"warnings": null
}
}
I've created a Kubernetes ServiceAccount with correct permissions as recommended in S3 repository | Elasticsearch Guide [8.6] | Elastic. I wonder why I get the above Access Denied error. Could you please help me out?