# Elasticsearch 7.16 use S3-compatible API setting return Failed to connect to service endpoint

**URL:** <https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [April 26, 2022, 3:36am UTC](https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208 "2022-04-26T03:36:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![forget6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forget6/32/104818_2.png) [@forget6](https://discuss.elastic.co/u/forget6)\
**Post date:** [April 26, 2022, 3:36am UTC](https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208/1 "2022-04-26T03:36:49Z")

</div>

# elasticsearch version: 7.16.2

s3 setting:

```auto
elasticsearch.yml
   s3.client.default.endpoint: ${our own s3}
bin/elasticsearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

```

================= commands:

```auto
PUT _snapshot/s3-test?error_trace
{
  "type": "s3",
  "settings": {
    "bucket": ${our own s3},
    "client": "default",
    "endpoint": ${our own s3 not aws},
    "path_style_access": "false"
  }
}

```

err:

```auto
       "type" : "repository_verification_exception",
        "reason" : "[s3-test] path is not accessible on master node",
        "stack_trace" : "RepositoryVerificationException[[s3-test] path is not accessible on master node]; nested: IOException[Unable to upload object [tests-xE9qIVoPSbKe-wUzA9qGVw/master.dat] using a single upload]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Failed to connect to service endpoint:]; nested: IOException[Connect timed out];\nCaused by: java.io.IOException: Unable to upload object [tests-xE9qIVoPSbKe-wUzA9qGVw/master.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:492)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:263)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1677)\n\tat org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: NotSerializableExceptionWrapper[sdk_client_exception: Failed to connect to service endpoint:]; nested: IOException[Connect timed out];\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:490)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:490)\n\t... 13 more\nCaused by: java.io.IOException: Connect timed out\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)\n\tat java.net.Socket.connect(Socket.java:633)\n\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\tat 
com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t... 45 more\n"

```

why? is setting not effect？

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 26, 2022, 4:06am UTC](https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208/2 "2022-04-26T04:06:04Z")

</div>

Welcome to our community! 😃

> [@forget6](#):
>
> `s3.client.default.endpoint: ${our own s3}`

What service are you using?

---

<div class="post-metadata">

**Author:** ![forget6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forget6/32/104818_2.png) [@forget6](https://discuss.elastic.co/u/forget6)\
**Post date:** [April 26, 2022, 7:56am UTC](https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208/5 "2022-04-26T07:56:04Z")

</div>

our s3 team reply: no flow to s3 from my IP，i guess the setting maybe not effect，how can i confirm it.[quote="warkolm, post:2, topic:303208, full:true"]  
Welcome to our community! 😃

> [@forget6](#):
>
> `s3.client.default.endpoint: ${our own s3}`

What service are you using?  
[/quote]

our company s3, our s3 team reply: no flow from my ip to s3

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 26, 2022, 8:32am UTC](https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208/6 "2022-04-26T08:32:52Z")

</div>

```nohighlight
Caused by: NotSerializableExceptionWrapper[sdk_client_exception: Failed to connect to service endpoint:]; nested: IOException[Connect timed out];
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)
	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)
	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
	at java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)
	at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)

```

This indicates that Elasticsearch is trying (and failing) to get instance credentials from the instance metadata service, which means it does not see the access key and secret key in the keystore. If you configured your keystore after starting Elasticsearch you will need to [reload secure settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-reload-secure-settings.html). Also make sure the keystore is in the correct location.

---

<div class="post-metadata">

**Author:** ![forget6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forget6/32/104818_2.png) [@forget6](https://discuss.elastic.co/u/forget6)\
**Post date:** [April 26, 2022, 9:04am UTC](https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208/9 "2022-04-26T09:04:17Z")

</div>

i have restarted server.but it still cannot work. my config is a new dir.i use export to redirect it.

```auto
ES_DIR=/data/ins_1
ES_HOME=/data/es/elasticsearch
ES_PATH_CONF=${ES_DIR}/config
export ES_PATH_CONF

```

i cp all config file in this new dir /data/ins\_1/config. but i find after i run

```auto
elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key

```

the config file Elasticsearch.keystore is not same as my config dir.

```auto
cd /data/ins_1/config
md5sum elasticsearch.keystore 
3b6c1e50190823cf91dca7191293097d elasticsearch.keystore

cd /data/es/elasticsearch/config
md5sum elasticsearch.keystore 
3e20d606979ac6c17123895b9fae32b0 elasticsearch.keystore

```

Elasticsearch-keystore add only change file in defalut dir?  
the keystore only save in Elasticsearch.keystore?  
if i cp the Elasticsearch.keystore to my config dir can solve it?  
how can i know which Elasticsearch.keystore is using?

---

<div class="post-metadata">

**Author:** ![forget6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forget6/32/104818_2.png) [@forget6](https://discuss.elastic.co/u/forget6)\
**Post date:** [April 26, 2022, 9:32am UTC](https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208/10 "2022-04-26T09:32:20Z")

</div>

i solve it. the keystore is wrong. thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 24, 2022, 9:32am UTC](https://discuss.elastic.co/t/elasticsearch-7-16-use-s3-compatible-api-setting-return-failed-to-connect-to-service-endpoint/303208/11 "2022-05-24T09:32:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.