Authentication issue with S3 Repository in EKS

Elastic Stack Elasticsearch
1

I am having an issue attempting to setup a S3 snapshot repository for a cluster running in EKS.

The serviceaccount is setup to use a role as detailed in https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html. I am able to log into a pod and manually execute s3 commands without issue so I believe the role is working correctly. I have validated that removing or changing the role restricts access as expected.

However, when creating/verifying an S3 repository it doesn't work. It almost seems like the service account role isn't being used. Is there something else I need to do to get elastic to recognize the service role?

Any ideas?

Thanks -andy

Creating the respository:

PUT /_snapshot/s3-repo
{
  "type": "s3",
  "settings": {
    "bucket": "asipe-es-backup-test"
  }
}

Output:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3-repo] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[s3-repo] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-PBsgxnX2ThmT6dvpss12CA/master.dat] using a single upload",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3387CEC20182CC41; S3 Extended Request ID: ZHGYwkKdwWfWzdZJT9Hw5NrQxPnpdBwrUTPRsqUZVYPIC3gWFya2uE7M9XzifIWc/zujT/k7ADA=)"
      }
    }
  },
  "status" : 500
}

Logs:

{"type": "server", "timestamp": "2020-03-27T12:58:05,241Z", "level": "INFO", "component": "o.e.r.RepositoriesService", "cluster.name": "asipe-es-quickstart", "node.name": "asipe-es-quickstart-es-default-0", "message": "put repository [s3-repo]", "cluster.uuid": "HcnS7rvzRq25PC5jnSqNrA", "node.id": "PwYg1EdbQmuhnf8F50nS3w"  }
{"type": "server", "timestamp": "2020-03-27T12:58:05,336Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "asipe-es-quickstart", "node.name": "asipe-es-quickstart-es-default-0", "message": "path: /_snapshot/s3-repo, params: {pretty=true, repository=s3-repo}", "cluster.uuid": "HcnS7rvzRq25PC5jnSqNrA", "node.id": "PwYg1EdbQmuhnf8F50nS3w" ,
"stacktrace": ["org.elasticsearch.repositories.RepositoryVerificationException: [s3-repo] path  is not accessible on master node",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1041) ~[elasticsearch-7.6.1.jar:7.6.1]",
.....
"Caused by: java.io.IOException: Unable to upload object [tests-PBsgxnX2ThmT6dvpss12CA/master.dat] using a single upload",
"at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:323) ~[?:?]",
.......
"... 6 more",
"Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3387CEC20182CC41; S3 Extended Request ID: ZHGYwkKdwWfWzdZJT9Hw5NrQxPnpdBwrUTPRsqUZVYPIC3gWFya2uE7M9XzifIWc/zujT/k7ADA=)",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) ~[?:?]",
2

If I specifically set access and secret keys in the key store for the default client then the repository works as expected.

Using the profile assigned to the service account would be much preferred. Is this supported?

3

Any ideas?

4

This appears to be related to this issue: https://github.com/elastic/elasticsearch/pull/53962

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.