# Snapshot to S3 error

**URL:** <https://discuss.elastic.co/t/snapshot-to-s3-error/362547>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [July 4, 2024, 12:26pm UTC](https://discuss.elastic.co/t/snapshot-to-s3-error/362547 "2024-07-04T12:26:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![koskartsev](https://avatars.discourse-cdn.com/v4/letter/k/b9bd4f/32.png) [@koskartsev](https://discuss.elastic.co/u/koskartsev)\
**Post date:** [July 4, 2024, 12:26pm UTC](https://discuss.elastic.co/t/snapshot-to-s3-error/362547/1 "2024-07-04T12:26:13Z")

</div>

Hi,

Elasticsearch 8.2.3  
I'm trying to add an S3 snapshot repository, the endpoint is Yandexcloud S3

I've set the keys in the keystore on all master nodes, check it with show

```auto
elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key

```

Then I add the snapshot repository:

```auto
PUT _snapshot/ycs3
{
 "type": "s3",
 "settings": {
   "bucket": "es-test",
   "endpoint": "storage.yandexcloud.net",
   "client": "default",
   "base_path": "path",
   "canned_acl": "bucket-owner-full-control"
 }
}

```

Which returnes error:

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[ycs3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]",
        "suppressed" : [
          {
            "type" : "repository_verification_exception",
            "reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]"
          },
          {
            "type" : "repository_verification_exception",
            "reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]"
          }
        ]
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[ycs3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]",
    "suppressed" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unable to upload object [path/tests-UY6djZJTQse3B1gdOrwmpA/data-PU4UCt3dR5S6_r8LuMhNLA.dat] using a single upload",
          "caused_by" : {
            "type" : "sdk_client_exception",
            "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
            "caused_by" : {
              "type" : "i_o_exception",
              "reason" : "Network is unreachable"
            }
          }
        }
      },
      {
        "type" : "repository_verification_exception",
        "reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unable to upload object [path/tests-UY6djZJTQse3B1gdOrwmpA/data-QLd6TNjhQHWKNgLpm3PzFg.dat] using a single upload",
          "caused_by" : {
            "type" : "sdk_client_exception",
            "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
            "caused_by" : {
              "type" : "i_o_exception",
              "reason" : "Network is unreachable"
            }
          }
        }
      }
    ]
  },
  "status" : 500
}

```

I see a folder with files created by the ES in the bucket  
`path/tests-UY6djZJTQse3B1gdOrwmpA`  
Then i take snapshot, got error:

```auto
INTERNAL_SERVER_ERROR: IndexShardSnapshotFailedException[Failed to write shard level snapshot metadata for [snapshot-name-4bb7krxwtk6hp_v3ovsz5a/ROb80jvRR0WH2DNmdbUnfg] to [index-_14V5serSM6Ub6UM79c1Nw]]; nested: IOException[Unable to upload object [path/indices/HEDN0S9lTm-iQZm9bKGxiQ/0/index-_14V5serSM6Ub6UM79c1Nw] using a single upload]; nested: SdkClientException[Failed to connect to service endpoint:]; nested: SocketException[Network is unreachable]

```

path/indexes/HEDN0S9lTm-iQZm9bKGxiQ/meta-.dat file appears in the bucket

S3 ACL is fine, i can read write to a bucket

---

<div class="post-metadata">

**Author:** ![koskartsev](https://avatars.discourse-cdn.com/v4/letter/k/b9bd4f/32.png) [@koskartsev](https://discuss.elastic.co/u/koskartsev)\
**Post date:** [July 4, 2024, 1:00pm UTC](https://discuss.elastic.co/t/snapshot-to-s3-error/362547/2 "2024-07-04T13:00:50Z")

</div>

Request with ?error\_trace  
PUT \_snapshot/s3?

```auto
{
 "type": "s3",
 "settings": {
   "bucket": "estest",
   "endpoint": "storage.yandexcloud.net",
   "client": "default",
   "base_path": "path",
   "canned_acl": "bucket-owner-full-control"
 }
}

```

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]",
        "stack_trace" : "org.elasticsearch.repositories.RepositoryVerificationException: [s3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]\n\tSuppressed: org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]\n\tCaused by: org.elasticsearch.repositories.RepositoryVerificationException: [s3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]\n\tCaused by: java.io.IOException: Unable to upload object [path/tests-nZeR3PAuRXq5RzFZZ0kaCQ/data-PU4UCt3dR5S6_r8LuMhNLA.dat] using a single upload\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:486)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\t\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\t\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3207)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:134)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:161)\n\t\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\t\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287)\n\t\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)\n\t\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\t\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\t\tat java.lang.Thread.run(Thread.java:833)\n\tCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\t\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\t\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\t\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:270)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\t\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:484)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:484)\n\t\t... 16 more\n\tCaused by: java.io.IOException: Network is unreachable\n\t\tat sun.nio.ch.Net.pollConnect(Native Method)\n\t\tat sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\t\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\t\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594)\n\t\tat java.net.Socket.connect(Socket.java:633)\n\t\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\t\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\t\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t\t... 48 more\n\tSuppressed: org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]\n\tCaused by: org.elasticsearch.repositories.RepositoryVerificationException: [s3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]\n\tCaused by: java.io.IOException: Unable to upload object [path/tests-nZeR3PAuRXq5RzFZZ0kaCQ/data-QLd6TNjhQHWKNgLpm3PzFg.dat] using a single upload\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:486)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\t\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\t\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3207)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:134)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:161)\n\t\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\t\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287)\n\t\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)\n\t\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\t\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\t\tat java.lang.Thread.run(Thread.java:833)\n\tCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\t\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\t\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\t\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:270)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\t\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:484)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:484)\n\t\t... 16 more\n\tCaused by: java.io.IOException: Network is unreachable\n\t\tat sun.nio.ch.Net.pollConnect(Native Method)\n\t\tat sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\t\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\t\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594)\n\t\tat java.net.Socket.connect(Socket.java:633)\n\t\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\t\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\t\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t\t... 48 more\n",
        "suppressed" : [
          {
            "type" : "repository_verification_exception",
            "reason" : "[s3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]",
            "stack_trace" : "org.elasticsearch.repositories.RepositoryVerificationException: [s3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]\nCaused by: java.io.IOException: Unable to upload object [path/tests-nZeR3PAuRXq5RzFZZ0kaCQ/data-PU4UCt3dR5S6_r8LuMhNLA.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:486)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3207)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:134)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:161)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:270)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:484)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:484)\n\t... 16 more\nCaused by: java.io.IOException: Network is unreachable\n\tat sun.nio.ch.Net.pollConnect(Native Method)\n\tat sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594)\n\tat java.net.Socket.connect(Socket.java:633)\n\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t... 48 more\n"
          },
          {
            "type" : "repository_verification_exception",
            "reason" : "[s3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]",
            "stack_trace" : "org.elasticsearch.repositories.RepositoryVerificationException: [s3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]\nCaused by: java.io.IOException: Unable to upload object [path/tests-nZeR3PAuRXq5RzFZZ0kaCQ/data-QLd6TNjhQHWKNgLpm3PzFg.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:486)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3207)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:134)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:161)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:270)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:484)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:484)\n\t... 16 more\nCaused by: java.io.IOException: Network is unreachable\n\tat sun.nio.ch.Net.pollConnect(Native Method)\n\tat sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594)\n\tat java.net.Socket.connect(Socket.java:633)\n\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t... 48 more\n"
          }
        ]
      }
    ]
}

```

---

<div class="post-metadata">

**Author:** ![koskartsev](https://avatars.discourse-cdn.com/v4/letter/k/b9bd4f/32.png) [@koskartsev](https://discuss.elastic.co/u/koskartsev)\
**Post date:** [July 4, 2024, 2:18pm UTC](https://discuss.elastic.co/t/snapshot-to-s3-error/362547/3 "2024-07-04T14:18:02Z")

</div>

Oh, i found missconfig

```auto
elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key

```

must be on **every** node, not only master