Verify snapshot repository 401 Unauthorized error

I am running ES (3 master, 3 data, and 3 ingest nodes) and Kibana on EKS using the ECK operator. I am trying to setup an S3 based snapshot repository but I receive a 401 Unauthorized error when trying to verify the repository in Kibana. I have configured everything based on the docs thus far.

  • Created the S3 bucket with recommended policy.
  • Created an IAM role that has access to the bucket.
  • Manually modified the default Kubernetes service account which the Elasticsearch node pods use with the IAM role annotation.
  • Verified all the pods are assuming the IAM role which has access to the bucket.
  • Verified access by connecting to a master node pod and uploaded/downloaded a file to the bucket using the awscli.
  • Created the directory /usr/share/elasticsearch/config/repository-s3 and a symlink of $AWS_WEB_IDENTITY_TOKEN_FILE to aws-web-identity-token-file in the same directory on all nodes. Did this after the pods were started.
  • Registered the snapshot repository in the Kibana UI using the correct bucket name and all the default settings.

When I click on the repository and then Verify repository, I get the following error. This happens with any bucket name I use. Any help would be appreciated.

{
  "name": "ResponseError",
  "meta": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "repository_verification_exception",
            "reason": "[es-development-backups] path  is not accessible on master node"
          }
        ],
        "type": "repository_verification_exception",
        "reason": "[es-development-backups] path  is not accessible on master node",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unable to upload object [tests-xTSezt2VQze-UcMp3Vv3rA/master.dat] using a single upload",
          "caused_by": {
            "type": "amazon_service_exception",
            "reason": "amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)"
          }
        }
      },
      "status": 500
    },
    "statusCode": 500,
    "headers": {
      "x-opaque-id": "0aa85525-497f-47a6-b2bd-45777cfb9e45;kibana:application:management:",
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json;charset=utf-8",
      "content-length": "594"
    },
    "meta": {
      "context": null,
      "request": {
        "params": {
          "method": "POST",
          "path": "/_snapshot/es-development-backups/_verify",
          "querystring": "",
          "headers": {
            "user-agent": "Kibana/8.5.1",
            "x-elastic-product-origin": "kibana",
            "authorization": "Basic REDACTED",
            "x-opaque-id": "0aa85525-497f-47a6-b2bd-45777cfb9e45;kibana:application:management:",
            "x-elastic-client-meta": "es=8.3.0p,js=16.18.1,t=8.2.0,hc=16.18.1",
            "accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
          }
        },
        "options": {
          "opaqueId": "0aa85525-497f-47a6-b2bd-45777cfb9e45;kibana:application:management:",
          "headers": {
            "x-elastic-product-origin": "kibana",
            "user-agent": "Kibana/8.5.1",
            "authorization": "Basic REDACTED",
            "x-opaque-id": "0aa85525-497f-47a6-b2bd-45777cfb9e45",
            "x-elastic-client-meta": "es=8.3.0p,js=16.18.1,t=8.2.0,hc=16.18.1"
          }
        },
        "id": 1
      },
      "name": "elasticsearch-js",
      "connection": {
        "url": "https://mycluster-dev-es-http.elastic-system.svc:9200/",
        "id": "https://mycluster-dev-es-http.elastic-system.svc:9200/",
        "headers": {},
        "status": "alive"
      },
      "attempts": 0,
      "aborted": false
    },
    "warnings": null
  }
}

Stack trace from one of the data nodes.

org.elasticsearch.transport.RemoteTransportException: [mycluster-dev-es-zone-a-master-0][100.72.216.125:9300][cluster:admin/repository/verify]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [es-development-backups] path  is not accessible on master node
Caused by: java.io.IOException: Unable to upload object [tests-XPt1021UQ5SubSch0-r_GQ/master.dat] using a single upload
	at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:485)
	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:127)
	at java.security.AccessController.doPrivileged(AccessController.java:569)
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)
	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:125)
	at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:119)
	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:259)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1680)
	at org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:481)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:892)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1589)
Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)
	at com.amazonaws.internal.EC2ResourceFetcher.handleErrorResponse(EC2ResourceFetcher.java:149)
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:94)
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:60)
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:48)
	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124)
	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80)
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
	at java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)
	at org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:276)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6431)
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6404)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5441)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)
	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)
	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:483)
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)
	at java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)
	at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:483)
	... 13 more

Welcome to our community! :smiley:

What version of Elasticsearch are you running?

Hi Mark. Thank you. I am running version 8.5.1.