S3 Access Denied error while verifying repository for snapshot and restore

[shawnmin]

Hi, I am working on Snapshot and Restore to ensure that my Elasticsearch indices are securely backed-up and stored. I've set up an EFK stack on my AWS EKS cluster. I've deployed the stack using Helm in Bitnami chart. I am using the recent version (8.6.2) of Elasticsearch and Kibana.

I've created an AWS S3 bucket with a name, say, my-eks-cluster-logging-snapshot-repository, and registered a repository using Kibana UI. Here are the details of the registered repository.

Type: AWS S3
Client: default
Bucket: my-eks-cluster-logging-snapshot-repository
Base path: Left as blank
Snapshot compression: Yes
Chunk size: Left as blank
Buffer size: Left as blank
Canned ACL: private
Storage class: standard
Max snapshot bytes per second: Left as blank
Max restore bytes per second: Left as blank
Read-only: No

I want to use IAM roles for Kubernetes service accounts for authentication, so I've checked the documentation about it in this link.

I've created repository-s3 directory in the Elasticsearch configuration path of elasticsearch-master nodes, and have created a symbolic link as said in the documentation.

However, when I click on Verify repository in Kibana UI, it fails and shows following error message.

{
  "name": "ResponseError",
  "meta": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "repository_verification_exception",
            "reason": "[my-eks-cluster-logging-snapshot-repository] path  is not accessible on master node"
          }
        ],
        "type": "repository_verification_exception",
        "reason": "[my-eks-cluster-logging-snapshot-repository] path  is not accessible on master node",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unable to upload object [tests-<REDACTED>/master.dat] using a single upload",
          "caused_by": {
            "type": "amazon_s3_exception",
            "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <REDACTED>; S3 Extended Request ID: <REDACTED>/<REDACTED>/<REDACTED>=; Proxy: null)"
          }
        }
      },
      "status": 500
    },
    "statusCode": 500,
    "headers": {
      "x-opaque-id": "bb3a5e7c-dea5-4b31-9b6c-<REDACTED>;kibana:application:management:",
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json;charset=utf-8",
      "content-length": "764"
    },
    "meta": {
      "context": null,
      "request": {
        "params": {
          "method": "POST",
          "path": "/_snapshot/my-eks-cluster-logging-snapshot-repository/_verify",
          "querystring": "",
          "headers": {
            "user-agent": "Kibana/8.6.2",
            "x-elastic-product-origin": "kibana",
            "x-opaque-id": "bb3a5e7c-dea5-4b31-9b6c-<REDACTED>;kibana:application:management:",
            "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1",
            "accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
          }
        },
        "options": {
          "opaqueId": "bb3a5e7c-dea5-4b31-9b6c-<REDACTED>;kibana:application:management:",
          "headers": {
            "x-elastic-product-origin": "kibana",
            "user-agent": "Kibana/8.6.2",
            "x-opaque-id": "bb3a5e7c-dea5-4b31-9b6c-<REDACTED>",
            "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"
          }
        },
        "id": 1
      },
      "name": "elasticsearch-js",
      "connection": {
        "url": "http://elasticsearch:9200/",
        "id": "http://elasticsearch:9200/",
        "headers": {},
        "status": "alive"
      },
      "attempts": 0,
      "aborted": false
    },
    "warnings": null
  }
}

I've created a Kubernetes ServiceAccount with correct permissions as recommended in S3 repository | Elasticsearch Guide [8.6] | Elastic. I wonder why I get the above Access Denied error. Could you please help me out?