S3 Repo verification exception (Failed to connect to service endpoint)

Elastic Stack Elasticsearch
1

Hello
I'm using snapshotting for backup/restore.
There I recreate each time the repository since the location in S3 can change.
For the authentication I use two different clients "backup" and "restore". At the moment access_key and secret_key for both clients are the same.
I use cerebro to create the S3 repository manually.
If I use the backup client the repo creation is successful but with exact the same request but using the restore client the repo creation fails with following error:

RepositoryVerificationException: Unable to upload object [tests-WIK_URQHS16ZMgG8K9fuYQ/master.dat]
Caused by: com.amazonaws.SdkClientException: Failed to connect to service endpoint:
java.net.SocketTimeoutException: Read timed out

Sometimes I also get the error:

"caused_by": { 
"type": "amazon_service_exception",
"reason": "Gateway Timeout (Service: null; Status Code: 504; Error Code: null; Request ID: null; Proxy: null)"

At the moment I don't have any idea how that can happen since the two client contains the same credentials.
Maybe as information, there is only one elasticsearch node running in the cluster.

Has somebody an idea what the problem could be?

Thanks Thomas

2

Hello @staubt

Welcome back!!

Could you please share the ELK version along with successful & unsuccessful request.
For the unsuccessful request share the complete response details.

Thanks!!

3

Hi
Thanks for the response. I'm using Elasticsearch 8.18.1. I don't saw this issue in the older version 8.15.3.

The successful request looks like:

PUT _snapshot/blob_backup_chsi
{
  "type": "s3",
  "settings": {
    "bucket": "ae-ace-sr28-dpl-1234-backups",
	"client": "backup",
	"basePath": "backups/2025-06-15T21-15-28",
	"region": "eu-north-1",
	"location": "",
	"compress": true
  }
}

The failing one is completly the same but using "client": "restore"

Thats what I found in the elasticsearch log

2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|org.elasticsearch.repositories.RepositoryVerificationException: [blob_backup_chsi] path  is not accessible on master node
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|Caused by: java.io.IOException: Unable to upload object [tests-WIK_URQHS16ZMgG8K9fuYQ/master.dat] using a single upload
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:466) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2211) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|Caused by: com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:60) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:48) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:32) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:318) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464) ~[?:?]
2025-06-16 09:26:35,852|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2211) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|Caused by: java.net.SocketTimeoutException: Read timed out
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.net.Socket$SocketInputStream.read(Socket.java:1109) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.io.BufferedInputStream.fill(BufferedInputStream.java:291) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.io.BufferedInputStream.read1(BufferedInputStream.java:347) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.io.BufferedInputStream.implRead(BufferedInputStream.java:420) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.io.BufferedInputStream.read(BufferedInputStream.java:399) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1708) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1617) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:82) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:60) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:48) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:32) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:318) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861) ~[?:?]
2025-06-16 09:26:35,853|INFO |Thread-1| MAIN_CONSOLE - /|	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2211) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.1.jar:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
2025-06-16 09:26:35,854|INFO |Thread-1| MAIN_CONSOLE - /|	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
4

The connect failure is happening when trying to contact IMDS, which only happens if you have not configured any other credentials for this repository. If it works with client: backup but not client: restore then I expect you have configured s3.client.backup.access_key and s3.client.backup.secret_key in your keystore but have not configured s3.client.restore.access_key or s3.client.restore.secret_key.

5

Hello David

Thanks a lot for your answer.
In the past we had from time to time the issue that the clients were not configured. If that was the case we got the exception that the given client does not exist with a list of existing client (in that case often only the default client).
f.e.

"caused_by": { - 
      "type": "illegal_argument_exception",
      "reason": "Unknown s3 client name [restore]. Existing client configs: backup,default"
    }

Yesterday I checked the keystore multiple times and could not find a difference.
But after my current meeting I will check again and also try to recreate the restore client with executing reload security settings.

6

You would get this message if you have no settings anywhere matching s3.client.restore.*, so its absence means that some such setting is configured. That does not imply that you have configured both s3.client.restore.access_key and s3.client.restore.secret_key in your keystore.