Need help configuring repository-s3 (AWS_WEB_IDENTITY_TOKEN_FILE ignored?)

Hello,

I'm having some difficulties setting up the repository-s3 plugin on Kubernetes (EKS) using ECK.

I can confirm that my service account is configured correctly, I can launch a pod running ubuntu with this service account in the same namespace, and I can list and put files in my bucket using the AWS command line.

It seems to just ignore my AWS_WEB_IDENTITY_TOKEN_FILE .

I have set up the symlink as it says in the documentation. All my nodes have this init container:

        - name: setup-s3-plugin
          env:
          - name: ES_PATH_CONF
            value: /usr/share/elasticsearch/config
          command:
          - sh
          - -c
          - |
            mkdir -p "${ES_PATH_CONF}/repository-s3" && ln -vs $AWS_WEB_IDENTITY_TOKEN_FILE "${ES_PATH_CONF}/repository-s3/aws-web-identity-token-file"

And all my nodes are correctly launching pods with the service account. The Elasticsearch user can read the symlink.

Here's the payload that I send:

PUT _snapshot/eric_s3_repository
{
  "type": "s3",
  "settings": {
    "bucket": "elasticsearch-poc"
  }
}

Then it fails about a timeout error connecting to an endpoint (that looks empty?) ; It doesn't look like a timeout though, the error happens really quickly.

I've tried different settings in Elasticsearch.yml regarding the s3 client, the endpoint, the proxy host and url, the region, etc. always the same result.

Any help would be appreciated.

Here's the full stack trace of the error:

org.elasticsearch.transport.RemoteTransportException: [elasticsearch-poc-es-default-3][172.19.10.187:9300][cluster:admin/repository/put]
     Caused by: org.elasticsearch.repositories.RepositoryException: [eric_s3_repository] Could not determine repository generation from root blobs
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1907) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:833) [?:?]
     Caused by: java.io.IOException: Exception when listing blobs by prefix [index-]
     at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:400) ~[?:?]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:833) ~[?:?]
     Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint: 
     at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
     at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
     at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
     at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
     at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
     at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
     at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
     at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
     at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
     at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
     at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442) ~[?:?]
     at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
     at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395) ~[?:?]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:833) ~[?:?]
     Caused by: java.io.IOException: Connect timed out
     at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
     at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
     at java.net.Socket.connect(Socket.java:633) ~[?:?]
     at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
     at sun.net.www.http.HttpClient.openServer(HttpClient.java:498) ~[?:?]
     at sun.net.www.http.HttpClient.openServer(HttpClient.java:603) ~[?:?]
     at sun.net.www.http.HttpClient.<init>(HttpClient.java:246) ~[?:?]
     at sun.net.www.http.HttpClient.New(HttpClient.java:351) ~[?:?]
     at sun.net.www.http.HttpClient.New(HttpClient.java:373) ~[?:?]
     at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309) ~[?:?]
     at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287) ~[?:?]
     at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]
     at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057) ~[?:?]
     at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
     at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
     at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
     at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
     at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
     at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
     at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
     at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
     at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
     at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
     at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
     at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994) ~[?:?]
     at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442) ~[?:?]
     at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
     at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442) ~[?:?]
     at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395) ~[?:?]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.16.3.jar:7.16.3]
     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.16.3.jar:7.16.3]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
     at java.lang.Thread.run(Thread.java:833) ~[?:?]

I am running ES version 7.16.3

I believe you need to upgrade to 8.x for this to work:

Thanks David!

I just upgraded to 8.2.2 and it does work!

I ran into an issue that I'll write here just in case anyone else run into the same problem:

Our EKS cluster does not have access to the internet, so it couldn't assume role (timeout connecting to sts.amazonaws.com). The s3.client.default.proxy.* parameters seems to be ignored when trying to connect to sts; But setting up the HTTP(S)_PROXY env variables did the trick.

Thanks!