Need help configuring repository-s3 (AWS_WEB_IDENTITY_TOKEN_FILE ignored?)

ericsperano · June 7, 2022, 11:30pm

Hello,

I'm having some difficulties setting up the repository-s3 plugin on Kubernetes (EKS) using ECK.

I can confirm that my service account is configured correctly, I can launch a pod running ubuntu with this service account in the same namespace, and I can list and put files in my bucket using the AWS command line.

It seems to just ignore my AWS_WEB_IDENTITY_TOKEN_FILE .

I have set up the symlink as it says in the documentation. All my nodes have this init container:

        - name: setup-s3-plugin
          env:
          - name: ES_PATH_CONF
            value: /usr/share/elasticsearch/config
          command:
          - sh
          - -c
          - |
            mkdir -p "${ES_PATH_CONF}/repository-s3" && ln -vs $AWS_WEB_IDENTITY_TOKEN_FILE "${ES_PATH_CONF}/repository-s3/aws-web-identity-token-file"

And all my nodes are correctly launching pods with the service account. The Elasticsearch user can read the symlink.

Here's the payload that I send:

PUT _snapshot/eric_s3_repository
{
  "type": "s3",
  "settings": {
    "bucket": "elasticsearch-poc"
  }
}

Then it fails about a timeout error connecting to an endpoint (that looks empty?) ; It doesn't look like a timeout though, the error happens really quickly.

I've tried different settings in Elasticsearch.yml regarding the s3 client, the endpoint, the proxy host and url, the region, etc. always the same result.

Any help would be appreciated.

Here's the full stack trace of the error:

org.elasticsearch.transport.RemoteTransportException: [elasticsearch-poc-es-default-3][172.19.10.187:9300][cluster:admin/repository/put]
     Caused by: org.elasticsearch.repositories.RepositoryException: [eric_s3_repository] Could not determine repository generation from root blobs
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1907) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:833) [?:?]
     Caused by: java.io.IOException: Exception when listing blobs by prefix [index-]
     at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:400) ~[?:?]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:833) ~[?:?]
     Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint: 
     at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
     at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
     at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
     at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
     at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
     at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
     at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
     at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
     at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
     at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
     at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442) ~[?:?]
     at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
     at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395) ~[?:?]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:833) ~[?:?]
     Caused by: java.io.IOException: Connect timed out
     at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
     at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
     at java.net.Socket.connect(Socket.java:633) ~[?:?]
     at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
     at sun.net.www.http.HttpClient.openServer(HttpClient.java:498) ~[?:?]
     at sun.net.www.http.HttpClient.openServer(HttpClient.java:603) ~[?:?]
     at sun.net.www.http.HttpClient.<init>(HttpClient.java:246) ~[?:?]
     at sun.net.www.http.HttpClient.New(HttpClient.java:351) ~[?:?]
     at sun.net.www.http.HttpClient.New(HttpClient.java:373) ~[?:?]
     at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309) ~[?:?]
     at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287) ~[?:?]
     at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]
     at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057) ~[?:?]
     at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
     at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
     at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
     at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
     at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
     at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
     at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
     at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
     at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
     at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
     at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442) ~[?:?]
     at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
     at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395) ~[?:?]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:833) ~[?:?]

I am running ES version 7.16.3

DavidTurner · June 8, 2022, 7:29am

I believe you need to upgrade to 8.x for this to work:

github.com/elastic/elasticsearch

[s3-repository] Support IAM roles for Kubernetes service accounts

elastic:master ← arteam:amazon-web-identity-token

opened 02:56PM - 02 Dec 21 UTC

arteam

+774 -50

There have been many requests to support `repository-s3` authentication via [IAM… roles in Kubernetes service accounts](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/). The AWS SDK is supposed to support them out of the box with the `aws-java-sdk-sts` library. Unfortunately, we can't use `WebIdentityTokenCredentialsProvider` from the SDK. It reads the token from `AWS_WEB_IDENTITY_TOKEN_FILE` environment variable which is usually mounted to `/var/run/secrets/eks.amazonaws.com/serviceaccount/token` and the S3 repository doesn't have the permissions to read it. We don't want to hard-code a file permission for the S3 repository, because the location of `AWS_WEB_IDENTITY_TOKEN_FILE` can change at any time in the future and we would also generally prefer to restrict the ability of plugins to access things outside of their config directory. To overcome this limitation, this change adds a custom `WebIdentityCredentials` provider that reads the service account from a symlink to `AWS_WEB_IDENTITY_TOKEN_FILE` created in the repository's config directory. We expect the end user to create the symlink to indicate that they want to use service accounts for authentication like this: ```bash mkdir -p "${ES_PATH_CONF}/repository-s3" ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "${ES_PATH_CONF}/repository-s3/aws-web-identity-token-file" ``` Service accounts are checked and exchanged for session tokens by the AWS STS. To test the authentification flow, this change adds a test fixture which mocks the `assume-role-with-web-identity` call to the service and returns a response with test credentials. Fixes #52625

ericsperano · June 10, 2022, 12:00am

Thanks David!

I just upgraded to 8.2.2 and it does work!

I ran into an issue that I'll write here just in case anyone else run into the same problem:

Our EKS cluster does not have access to the internet, so it couldn't assume role (timeout connecting to sts.amazonaws.com). The s3.client.default.proxy.* parameters seems to be ignored when trying to connect to sts; But setting up the HTTP(S)_PROXY env variables did the trick.

Thanks!

system · July 8, 2022, 12:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Enabling repository-s3 plugin to backup ES snapshots to S3 Elastic Cloud on Kubernetes (ECK) snapshot-and-restore	3	840	November 23, 2021
Elasticsearch 8.3 intermittent snapshot issues with AWS S3 on EKS Elasticsearch docker , snapshot-and-restore	1	626	September 2, 2022
S3 Access Denied error while verifying repository for snapshot and restore Elasticsearch snapshot-and-restore	1	534	May 2, 2023
Verify snapshot repository 401 Unauthorized error Elastic Cloud on Kubernetes (ECK) snapshot-and-restore	4	550	June 8, 2023
Repository-s3 not uploading snapshot to s3 Elasticsearch	4	510	December 3, 2019

Need help configuring repository-s3 (AWS_WEB_IDENTITY_TOKEN_FILE ignored?)

Related topics