# Need help configuring repository-s3 (AWS\_WEB\_IDENTITY\_TOKEN\_FILE ignored?) **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [June 7, 2022, 11:30pm UTC](https://discuss.elastic.co/t/need-help-configuring-repository-s3-aws-web-identity-token-file-ignored/306607 "2022-06-07T23:30:31Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![ericsperano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericsperano/32/106739_2.png) [@ericsperano](https://discuss.elastic.co/u/ericsperano)\ **Post date:** [June 7, 2022, 11:30pm UTC](https://discuss.elastic.co/t/need-help-configuring-repository-s3-aws-web-identity-token-file-ignored/306607/1 "2022-06-07T23:30:31Z")
Hello, I'm having some difficulties setting up the repository-s3 plugin on Kubernetes (EKS) using ECK. I can confirm that my service account is configured correctly, I can launch a pod running ubuntu with this service account in the same namespace, and I can list and put files in my bucket using the AWS command line. It seems to just ignore my AWS\_WEB\_IDENTITY\_TOKEN\_FILE . I have set up the symlink as it says in the documentation. All my nodes have this init container: ```auto - name: setup-s3-plugin env: - name: ES_PATH_CONF value: /usr/share/elasticsearch/config command: - sh - -c - | mkdir -p "${ES_PATH_CONF}/repository-s3" && ln -vs $AWS_WEB_IDENTITY_TOKEN_FILE "${ES_PATH_CONF}/repository-s3/aws-web-identity-token-file" ``` And all my nodes are correctly launching pods with the service account. The Elasticsearch user can read the symlink. Here's the payload that I send: ```auto PUT _snapshot/eric_s3_repository { "type": "s3", "settings": { "bucket": "elasticsearch-poc" } } ``` Then it fails about a timeout error connecting to an endpoint (that looks empty?) ; It doesn't look like a timeout though, the error happens really quickly. I've tried different settings in Elasticsearch.yml regarding the s3 client, the endpoint, the proxy host and url, the region, etc. always the same result. Any help would be appreciated. Here's the full stack trace of the error: ```auto org.elasticsearch.transport.RemoteTransportException: [elasticsearch-poc-es-default-3][172.19.10.187:9300][cluster:admin/repository/put] Caused by: org.elasticsearch.repositories.RepositoryException: [eric_s3_repository] Could not determine repository generation from root blobs at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1907) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: java.io.IOException: Exception when listing blobs by prefix [index-] at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:400) ~[?:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint: at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?] at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?] at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?] at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?] at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?] at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?] at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?] at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?] at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?] at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?] at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895) ~[?:?] at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442) ~[?:?] at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?] at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?] at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442) ~[?:?] at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395) ~[?:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: java.io.IOException: Connect timed out at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?] at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?] at java.net.Socket.connect(Socket.java:633) ~[?:?] at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?] at sun.net.www.http.HttpClient.openServer(HttpClient.java:498) ~[?:?] at sun.net.www.http.HttpClient.openServer(HttpClient.java:603) ~[?:?] at sun.net.www.http.HttpClient.(HttpClient.java:246) ~[?:?] at sun.net.www.http.HttpClient.New(HttpClient.java:351) ~[?:?] at sun.net.www.http.HttpClient.New(HttpClient.java:373) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057) ~[?:?] at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?] at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?] at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?] at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?] at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?] at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?] at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?] at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?] at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?] at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?] at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?] at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994) ~[?:?] at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895) ~[?:?] at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442) ~[?:?] at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?] at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?] at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442) ~[?:?] at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395) ~[?:?] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] ``` I am running ES version 7.16.3 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 8, 2022, 7:29am UTC](https://discuss.elastic.co/t/need-help-configuring-repository-s3-aws-web-identity-token-file-ignored/306607/2 "2022-06-08T07:29:52Z")
> [@ericsperano](#): > > I am running ES version 7.16.3 I believe you need to upgrade to 8.x for this to work: > > > There have been many requests to support \`repository-s3\` authentication via \[IAM… roles in Kubernetes service accounts\](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/). > > The AWS SDK is supposed to support them out of the box with the \`aws-java-sdk-sts\` library. Unfortunately, we can't use \`WebIdentityTokenCredentialsProvider\` from the SDK. It reads the token from \`AWS\_WEB\_IDENTITY\_TOKEN\_FILE\` environment variable which is usually mounted to \`/var/run/secrets/eks.amazonaws.com/serviceaccount/token\` and the S3 repository doesn't have the permissions to read it. We don't want to hard-code a file permission for the S3 repository, because the location of \`AWS\_WEB\_IDENTITY\_TOKEN\_FILE\` can change at any time in the future and we would also generally prefer to restrict the ability of plugins to access things outside of their config directory. > > To overcome this limitation, this change adds a custom \`WebIdentityCredentials\` provider that reads the service account from a symlink to \`AWS\_WEB\_IDENTITY\_TOKEN\_FILE\` created in the repository's config directory. We expect the end user to create the symlink to indicate that they want to use service accounts for authentication like this: > > \`\`\`bash > mkdir -p "${ES\_PATH\_CONF}/repository-s3" > ln -s $AWS\_WEB\_IDENTITY\_TOKEN\_FILE "${ES\_PATH\_CONF}/repository-s3/aws-web-identity-token-file" > \`\`\` > > Service accounts are checked and exchanged for session tokens by the AWS STS. To test the authentification flow, this change adds a test fixture which mocks the \`assume-role-with-web-identity\` call to the service and returns a response with test credentials. > > Fixes #52625 ---
**Author:** ![ericsperano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericsperano/32/106739_2.png) [@ericsperano](https://discuss.elastic.co/u/ericsperano)\ **Post date:** [June 10, 2022, 12:00am UTC](https://discuss.elastic.co/t/need-help-configuring-repository-s3-aws-web-identity-token-file-ignored/306607/3 "2022-06-10T00:00:24Z")
Thanks David! I just upgraded to 8.2.2 and it does work! I ran into an issue that I'll write here just in case anyone else run into the same problem: Our EKS cluster does not have access to the internet, so it couldn't assume role (timeout connecting to [sts.amazonaws.com](http://sts.amazonaws.com)). The s3.client.default.proxy.\* parameters seems to be ignored when trying to connect to sts; But setting up the HTTP(S)\_PROXY env variables did the trick. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 8, 2022, 12:01am UTC](https://discuss.elastic.co/t/need-help-configuring-repository-s3-aws-web-identity-token-file-ignored/306607/4 "2022-07-08T00:01:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.