# S3 Repo verification exception (Failed to connect to service endpoint)

**URL:** <https://discuss.elastic.co/t/s3-repo-verification-exception-failed-to-connect-to-service-endpoint/379244>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [June 17, 2025, 1:00pm UTC](https://discuss.elastic.co/t/s3-repo-verification-exception-failed-to-connect-to-service-endpoint/379244 "2025-06-17T13:00:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![staubt](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@staubt](https://discuss.elastic.co/u/staubt)\
**Post date:** [June 17, 2025, 1:00pm UTC](https://discuss.elastic.co/t/s3-repo-verification-exception-failed-to-connect-to-service-endpoint/379244/1 "2025-06-17T13:00:29Z")

</div>

Hello  
I'm using snapshotting for backup/restore.  
There I recreate each time the repository since the location in S3 can change.  
For the authentication I use two different clients "backup" and "restore". At the moment access\_key and secret\_key for both clients are the same.  
I use cerebro to create the S3 repository manually.  
If I use the backup client the repo creation is successful but with exact the same request but using the restore client the repo creation fails with following error:

```auto
RepositoryVerificationException: Unable to upload object [tests-WIK_URQHS16ZMgG8K9fuYQ/master.dat]
Caused by: com.amazonaws.SdkClientException: Failed to connect to service endpoint:
java.net.SocketTimeoutException: Read timed out

```

Sometimes I also get the error:

```auto
"caused_by": { 
"type": "amazon_service_exception",
"reason": "Gateway Timeout (Service: null; Status Code: 504; Error Code: null; Request ID: null; Proxy: null)"

```

At the moment I don't have any idea how that can happen since the two client contains the same credentials.  
Maybe as information, there is only one elasticsearch node running in the cluster.

Has somebody an idea what the problem could be?

Thanks Thomas

---

<div class="post-metadata">

**Author:** ![Tortoise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tortoise/32/147587_2.png) [@Tortoise](https://discuss.elastic.co/u/Tortoise)\
**Post date:** [June 18, 2025, 4:34am UTC](https://discuss.elastic.co/t/s3-repo-verification-exception-failed-to-connect-to-service-endpoint/379244/2 "2025-06-18T04:34:51Z")

</div>

Hello @staubt

Welcome back!!

Could you please share the ELK version along with successful & unsuccessful request.  
For the unsuccessful request share the complete response details.

Thanks!!

---

<div class="post-metadata">

**Author:** ![staubt](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@staubt](https://discuss.elastic.co/u/staubt)\
**Post date:** [June 18, 2025, 5:26am UTC](https://discuss.elastic.co/t/s3-repo-verification-exception-failed-to-connect-to-service-endpoint/379244/3 "2025-06-18T05:26:41Z")

</div>

> [@Tortoise](#):
>
> ELK

Hi  
Thanks for the response. I'm using Elasticsearch 8.18.1. I don't saw this issue in the older version 8.15.3.

The successful request looks like:

```auto
PUT _snapshot/blob_backup_chsi
{
  "type": "s3",
  "settings": {
    "bucket": "ae-ace-sr28-dpl-1234-backups",
	"client": "backup",
	"basePath": "backups/2025-06-15T21-15-28",
	"region": "eu-north-1",
	"location": "",
	"compress": true
  }
}

```

The failing one is completly the same but using "client": "restore"

Thats what I found in the elasticsearch log

```auto
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|org.elasticsearch.repositories.RepositoryVerificationException: [blob_backup_chsi] path is not accessible on master node
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|Caused by: java.io.IOException: Unable to upload object [tests-WIK_URQHS16ZMgG8K9fuYQ/master.dat] using a single upload
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:466) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2211) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|Caused by: com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:60) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:48) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:32) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:318) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2211) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|Caused by: java.net.SocketTimeoutException: Read timed out
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.net.Socket$SocketInputStream.read(Socket.java:1109) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.io.BufferedInputStream.fill(BufferedInputStream.java:291) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.io.BufferedInputStream.read1(BufferedInputStream.java:347) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.io.BufferedInputStream.implRead(BufferedInputStream.java:420) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.io.BufferedInputStream.read(BufferedInputStream.java:399) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1708) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1617) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:82) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:60) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:48) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:32) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:318) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2211) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.lang.Thread.run(Thread.java:1583) ~[?:?]

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 18, 2025, 7:05am UTC](https://discuss.elastic.co/t/s3-repo-verification-exception-failed-to-connect-to-service-endpoint/379244/4 "2025-06-18T07:05:47Z")

</div>

> [@staubt](#):
>
> `InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint`

The connect failure is happening when trying to contact IMDS, which only happens if you have not configured any other credentials for this repository. If it works with `client: backup` but not `client: restore` then I expect you have configured `s3.client.backup.access_key` and `s3.client.backup.secret_key` in your keystore but have not configured `s3.client.restore.access_key` or `s3.client.restore.secret_key`.

---

<div class="post-metadata">

**Author:** ![staubt](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@staubt](https://discuss.elastic.co/u/staubt)\
**Post date:** [June 18, 2025, 7:57am UTC](https://discuss.elastic.co/t/s3-repo-verification-exception-failed-to-connect-to-service-endpoint/379244/5 "2025-06-18T07:57:02Z")

</div>

Hello David

Thanks a lot for your answer.  
In the past we had from time to time the issue that the clients were not configured. If that was the case we got the exception that the given client does not exist with a list of existing client (in that case often only the default client).  
f.e.

```auto
"caused_by": { - 
      "type": "illegal_argument_exception",
      "reason": "Unknown s3 client name [restore]. Existing client configs: backup,default"
    }

```

Yesterday I checked the keystore multiple times and could not find a difference.  
But after my current meeting I will check again and also try to recreate the restore client with executing reload security settings.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 18, 2025, 8:09am UTC](https://discuss.elastic.co/t/s3-repo-verification-exception-failed-to-connect-to-service-endpoint/379244/6 "2025-06-18T08:09:29Z")

</div>

> [@staubt](#):
>
> `Unknown s3 client name [restore].`

You would get this message if you have no settings _anywhere_ matching `s3.client.restore.*`, so its absence means that some such setting is configured. That does not imply that you have configured both `s3.client.restore.access_key` and `s3.client.restore.secret_key` in your keystore.