# Issue with creating backup to s3 **URL:** \ **Category:** Elasticsearch\ **Tags:** docker, snapshot-and-restore\ **Created:** [June 19, 2024, 6:52am UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701 "2024-06-19T06:52:36Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Vakh](https://avatars.discourse-cdn.com/v4/letter/v/85f322/32.png) [@Vakh](https://discuss.elastic.co/u/Vakh)\ **Post date:** [June 19, 2024, 6:52am UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/1 "2024-06-19T06:52:36Z")
Hello there! I will start by providing as many details as possible. First of all, I want to state that it is my first time working with Elastic and I do not know a lot. Now, in a company we have 3 EC2 instances, on which they are running Elasticsearch cluster, Elasticsearch is deployed as a docker container, and they need to create a backup for this Elasticsearch, here are the steps I have already done after googling up how to create a backup to s3 when Elasticsearch is running as a docker container, I created s3 bucket and installed s3 plugin using the following command: ```auto docker exec -it bin/elasticsearch-plugin install repository-s3 ``` I verified that the plugin is installed, after that I provided access and secret keys using the following commands: ```auto docker exec -it bin/elasticsearch-keystore add s3.client.default.access_key ``` ```auto docker exec -it bin/elasticsearch-keystore add s3.client.default.secret_key ``` the list shows that they are added, i made sure that this user has the required permissions and that credentials are new and updated, now here is the tricky part the following command returns with an error: ```auto docker exec -it elasticsearch curl -u elastic: password" -X PUT "localhost:9200/_snapshot/my_s3_repository" -H "Content-Type: application/json" -d '{ "type": "s3", "settings": { "bucket": ""bucketname"", "region": "region-name" } } ``` it returns: ```auto {"error":{"root_cause":[{"type":"repository_exception","reason":"[my_s3_repository] Could not determine repository generation from root blobs"}],"type":"repository_exception","reason":"[my_s3_repository] Could not determine repository generation from root blobs","caused_by":{"type":"i_o_exception","reason":"Exception when listing blobs by prefix [index-]","caused_by":{"type":"amazon_service_exception","reason":"amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)"}}},"status":500} ``` I am not able to understand how to resolve it, I also tried to check the connection with the next command: ```auto aws s3 ls s3://mybucket--region test-region ``` I installed and configured AWS CLI inside the container, provided the same access and secret key and was able to ls and see the files in the bucket. I am struggling to understand what step I am missing. Please advise. Thank you all in advance! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2024, 8:29am UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/2 "2024-06-19T08:29:50Z")
> [@Vakh](#): > > `amazon_service_exception: Unauthorized (Service: null; Status Code: 401` The access key and secret key you've provided apparently do not have sufficient permissions. See [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html#repository-s3-permissions) for more information. ---
**Author:** ![Vakh](https://avatars.discourse-cdn.com/v4/letter/v/85f322/32.png) [@Vakh](https://discuss.elastic.co/u/Vakh)\ **Post date:** [June 19, 2024, 8:51am UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/3 "2024-06-19T08:51:37Z")
Hello David! Thank you for your reply. I checked to make sure and AWS IAM auto user has administrator access and required permission to access s3 bucket, also as i mentioned i was able to list files in s3 after configuring aws cli with this very same keys. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2024, 9:22am UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/4 "2024-06-19T09:22:48Z")
I suspect `aws s3 ls ...` is doing something different from what Elasticsearch is doing. Could you try adding `?error_trace` to the URL so that it shares more details about exactly what's going wrong? I.e. say `localhost:9200/_snapshot/my_s3_repository?error_trace` instead of `localhost:9200/_snapshot/my_s3_repository`. ---
**Author:** ![Vakh](https://avatars.discourse-cdn.com/v4/letter/v/85f322/32.png) [@Vakh](https://discuss.elastic.co/u/Vakh)\ **Post date:** [June 19, 2024, 10:51am UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/5 "2024-06-19T10:51:13Z")
> [@DavidTurner](#): > > localhost:9200/\_snapshot/my\_s3\_repository?error\_trace Thank you for the suggestion, this is the new output for the error: ```auto {"error":{"root_cause":[{"type":"repository_exception","reason":"[my_s3_repository] Could not determine repository generation from root blobs","stack_trace":"RepositoryException[[my_s3_repository] Could not determine repository generation from root blobs]; nested: IOException[Exception when listing blobs by prefix [index-]]; nested: NotSerializableExceptionWrapper[amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1907)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: java.io.IOException: Exception when listing blobs by prefix [index-]\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:400)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904)\n\t... 6 more\nCaused by: NotSerializableExceptionWrapper[amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)]\n\tat com.amazonaws.internal.EC2ResourceFetcher.handleErrorResponse(EC2ResourceFetcher.java:149)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:94)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994)\n\tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395)\n\t... 9 more\n"}],"type":"repository_exception","reason":"[my_s3_repository] Could not determine repository generation from root blobs","caused_by":{"type":"i_o_exception","reason":"Exception when listing blobs by prefix [index-]","caused_by":{"type":"amazon_service_exception","reason":"amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)","stack_trace":"NotSerializableExceptionWrapper[amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)]\n\tat com.amazonaws.internal.EC2ResourceFetcher.handleErrorResponse(EC2ResourceFetcher.java:149)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:94)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994)\n\tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\n"},"stack_trace":"java.io.IOException: Exception when listing blobs by prefix [index-]\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:400)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: NotSerializableExceptionWrapper[amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)]\n\tat com.amazonaws.internal.EC2ResourceFetcher.handleErrorResponse(EC2ResourceFetcher.java:149)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:94)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994)\n\tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395)\n\t... 9 more\n"},"stack_trace":"RepositoryException[[my_s3_repository] Could not determine repository generation from root blobs]; nested: IOException[Exception when listing blobs by prefix [index-]]; nested: NotSerializableExceptionWrapper[amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1907)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: java.io.IOException: Exception when listing blobs by prefix [index-]\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:400)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904)\n\t... 6 more\nCaused by: NotSerializableExceptionWrapper[amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)]\n\tat com.amazonaws.internal.EC2ResourceFetcher.handleErrorResponse(EC2ResourceFetcher.java:149)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:94)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994)\n\tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395)\n\t... 9 more\n"},"status":500}[root@ip-10-68-116-211 centos]# ``` ---
**Author:** ![Vakh](https://avatars.discourse-cdn.com/v4/letter/v/85f322/32.png) [@Vakh](https://discuss.elastic.co/u/Vakh)\ **Post date:** [June 19, 2024, 1:16pm UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/6 "2024-06-19T13:16:50Z")
Thank you! I tried to add the error logs but it was seen as spam. I also tried to upload images but since I am a new user was not allowed to. Are there other ways I can share it? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2024, 1:25pm UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/7 "2024-06-19T13:25:45Z")
I see, sorry, the auto-spam thing can be a bit sensitive sometimes. I've retrieved your post from the bin 🙂 The problem is here: ```auto RepositoryException[[my_s3_repository] Could not determine repository generation from root blobs]; nested: IOException[Exception when listing blobs by prefix [index-]]; nested: NotSerializableExceptionWrapper[amazon_service_exception: Unautho at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1907) at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.lang.Thread.run(Thread.java:833) Caused by: java.io.IOException: Exception when listing blobs by prefix [index-] at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:400) at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2608) at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2580) at org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1904) ... 6 more Caused by: NotSerializableExceptionWrapper[amazon_service_exception: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)] at com.amazonaws.internal.EC2ResourceFetcher.handleErrorResponse(EC2ResourceFetcher.java:149) at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:94) at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) at java.security.AccessController.doPrivileged(AccessController.java:318) at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895) at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:442) at java.security.AccessController.doPrivileged(AccessController.java:318) at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:442) at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:395) ... 9 more ``` In other words, it's trying to get credentials from `InstanceMetadataServiceCredentialsFetcher`, which means it's not seeing the access key and secret key you (think you) added to the keystore. ---
**Author:** ![Vakh](https://avatars.discourse-cdn.com/v4/letter/v/85f322/32.png) [@Vakh](https://discuss.elastic.co/u/Vakh)\ **Post date:** [June 19, 2024, 1:50pm UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/8 "2024-06-19T13:50:37Z")
Thanks! Do you have any suggestions on how can we solve it? Or is there any workaround? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2024, 2:12pm UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/9 "2024-06-19T14:12:01Z")
Are you sure you put the keys in the right keystore? Docker can make it hard to pin this kind of thing down. Also did you reload the keystore (or restart the node) after setting those values? ---
**Author:** ![Vakh](https://avatars.discourse-cdn.com/v4/letter/v/85f322/32.png) [@Vakh](https://discuss.elastic.co/u/Vakh)\ **Post date:** [June 19, 2024, 2:46pm UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/10 "2024-06-19T14:46:35Z")
I checked again and yes, keys are stored in elasticsearch.keystore, i have restarted the elasticsearch service before, but to rule it out, i restarted the container again, but the error stays the same. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2024, 3:03pm UTC](https://discuss.elastic.co/t/issue-with-creating-backup-to-s3/361701/11 "2024-06-19T15:03:55Z")
Hm sorry something is wrong with how ES is configured, but I don't really know how to troubleshoot further given that Docker is involved.