Snapshot to S3 error

Hi,

Elasticsearch 8.2.3
I'm trying to add an S3 snapshot repository, the endpoint is Yandexcloud S3

I've set the keys in the keystore on all master nodes, check it with show

elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key

Then I add the snapshot repository:

PUT _snapshot/ycs3
{
 "type": "s3",
 "settings": {
   "bucket": "es-test",
   "endpoint": "storage.yandexcloud.net",
   "client": "default",
   "base_path": "path",
   "canned_acl": "bucket-owner-full-control"
 }
}

Which returnes error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[ycs3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]",
        "suppressed" : [
          {
            "type" : "repository_verification_exception",
            "reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]"
          },
          {
            "type" : "repository_verification_exception",
            "reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]"
          }
        ]
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[ycs3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]",
    "suppressed" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unable to upload object [path/tests-UY6djZJTQse3B1gdOrwmpA/data-PU4UCt3dR5S6_r8LuMhNLA.dat] using a single upload",
          "caused_by" : {
            "type" : "sdk_client_exception",
            "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
            "caused_by" : {
              "type" : "i_o_exception",
              "reason" : "Network is unreachable"
            }
          }
        }
      },
      {
        "type" : "repository_verification_exception",
        "reason" : "[ycs3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unable to upload object [path/tests-UY6djZJTQse3B1gdOrwmpA/data-QLd6TNjhQHWKNgLpm3PzFg.dat] using a single upload",
          "caused_by" : {
            "type" : "sdk_client_exception",
            "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
            "caused_by" : {
              "type" : "i_o_exception",
              "reason" : "Network is unreachable"
            }
          }
        }
      }
    ]
  },
  "status" : 500
}

I see a folder with files created by the ES in the bucket
path/tests-UY6djZJTQse3B1gdOrwmpA
Then i take snapshot, got error:

INTERNAL_SERVER_ERROR: IndexShardSnapshotFailedException[Failed to write shard level snapshot metadata for [snapshot-name-4bb7krxwtk6hp_v3ovsz5a/ROb80jvRR0WH2DNmdbUnfg] to [index-_14V5serSM6Ub6UM79c1Nw]]; nested: IOException[Unable to upload object [path/indices/HEDN0S9lTm-iQZm9bKGxiQ/0/index-_14V5serSM6Ub6UM79c1Nw] using a single upload]; nested: SdkClientException[Failed to connect to service endpoint: ]; nested: SocketException[Network is unreachable]

path/indexes/HEDN0S9lTm-iQZm9bKGxiQ/meta-.dat file appears in the bucket

S3 ACL is fine, i can read write to a bucket

Request with ?error_trace
PUT _snapshot/s3?

{
 "type": "s3",
 "settings": {
   "bucket": "estest",
   "endpoint": "storage.yandexcloud.net",
   "client": "default",
   "base_path": "path",
   "canned_acl": "bucket-owner-full-control"
 }
}
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]",
        "stack_trace" : "org.elasticsearch.repositories.RepositoryVerificationException: [s3] [[PU4UCt3dR5S6_r8LuMhNLA, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]'], [QLd6TNjhQHWKNgLpm3PzFg, 'org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]']]\n\tSuppressed: org.elasticsearch.transport.RemoteTransportException: [glb-general-esd02][192.168.248.86:9300][internal:admin/repository/verify]\n\tCaused by: org.elasticsearch.repositories.RepositoryVerificationException: [s3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]\n\tCaused by: java.io.IOException: Unable to upload object [path/tests-nZeR3PAuRXq5RzFZZ0kaCQ/data-PU4UCt3dR5S6_r8LuMhNLA.dat] using a single upload\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:486)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\t\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\t\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3207)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:134)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:161)\n\t\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\t\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287)\n\t\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)\n\t\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\t\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\t\tat java.lang.Thread.run(Thread.java:833)\n\tCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\t\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\t\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\t\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:270)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\t\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:484)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:484)\n\t\t... 16 more\n\tCaused by: java.io.IOException: Network is unreachable\n\t\tat sun.nio.ch.Net.pollConnect(Native Method)\n\t\tat sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\t\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\t\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594)\n\t\tat java.net.Socket.connect(Socket.java:633)\n\t\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\t\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\t\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t\t... 48 more\n\tSuppressed: org.elasticsearch.transport.RemoteTransportException: [glb-general-esd01][192.168.248.85:9300][internal:admin/repository/verify]\n\tCaused by: org.elasticsearch.repositories.RepositoryVerificationException: [s3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]\n\tCaused by: java.io.IOException: Unable to upload object [path/tests-nZeR3PAuRXq5RzFZZ0kaCQ/data-QLd6TNjhQHWKNgLpm3PzFg.dat] using a single upload\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:486)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\t\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\t\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3207)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:134)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\t\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:161)\n\t\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\t\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287)\n\t\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)\n\t\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\t\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\t\tat java.lang.Thread.run(Thread.java:833)\n\tCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)\n\t\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\t\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\t\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\t\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\t\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\t\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:270)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\t\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\t\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\t\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\t\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:484)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\t\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\t\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\t\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:484)\n\t\t... 16 more\n\tCaused by: java.io.IOException: Network is unreachable\n\t\tat sun.nio.ch.Net.pollConnect(Native Method)\n\t\tat sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\t\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\t\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594)\n\t\tat java.net.Socket.connect(Socket.java:633)\n\t\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\t\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\t\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\t\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\t\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\t\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\t\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t\t... 48 more\n",
        "suppressed" : [
          {
            "type" : "repository_verification_exception",
            "reason" : "[s3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]",
            "stack_trace" : "org.elasticsearch.repositories.RepositoryVerificationException: [s3] store location [estest] is not accessible on the node [{glb-general-esd02}{PU4UCt3dR5S6_r8LuMhNLA}{Vvgz_Ww3SAGqtNv6XP6pgg}{192.168.248.86}{192.168.248.86:9300}{d}{xpack.installed=true}]\nCaused by: java.io.IOException: Unable to upload object [path/tests-nZeR3PAuRXq5RzFZZ0kaCQ/data-PU4UCt3dR5S6_r8LuMhNLA.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:486)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3207)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:134)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:161)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:270)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:484)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:484)\n\t... 16 more\nCaused by: java.io.IOException: Network is unreachable\n\tat sun.nio.ch.Net.pollConnect(Native Method)\n\tat sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594)\n\tat java.net.Socket.connect(Socket.java:633)\n\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t... 48 more\n"
          },
          {
            "type" : "repository_verification_exception",
            "reason" : "[s3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]",
            "stack_trace" : "org.elasticsearch.repositories.RepositoryVerificationException: [s3] store location [estest] is not accessible on the node [{glb-general-esd01}{QLd6TNjhQHWKNgLpm3PzFg}{Edk6CIR8RXaSxF0CL-_pAg}{192.168.248.85}{192.168.248.85:9300}{d}{xpack.installed=true}]\nCaused by: java.io.IOException: Unable to upload object [path/tests-nZeR3PAuRXq5RzFZZ0kaCQ/data-QLd6TNjhQHWKNgLpm3PzFg.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:486)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:3207)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:134)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:166)\n\tat org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:161)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:270)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:484)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:484)\n\t... 16 more\nCaused by: java.io.IOException: Network is unreachable\n\tat sun.nio.ch.Net.pollConnect(Native Method)\n\tat sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594)\n\tat java.net.Socket.connect(Socket.java:633)\n\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t... 48 more\n"
          }
        ]
      }
    ]
}

Oh, i found missconfig

elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key

must be on every node, not only master