Elasticsearch 7.16 use S3-compatible API setting return Failed to connect to service endpoint

forget6 · April 26, 2022, 3:36am

Elasticsearch version: 7.16.2

s3 setting:

elasticsearch.yml
   s3.client.default.endpoint: ${our own s3}
bin/elasticsearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

================= commands:

PUT _snapshot/s3-test?error_trace
{
  "type": "s3",
  "settings": {
    "bucket": ${our own s3},
    "client": "default",
    "endpoint": ${our own s3 not aws},
    "path_style_access": "false"
  }
}

err:

       "type" : "repository_verification_exception",
        "reason" : "[s3-test] path  is not accessible on master node",
        "stack_trace" : "RepositoryVerificationException[[s3-test] path  is not accessible on master node]; nested: IOException[Unable to upload object [tests-xE9qIVoPSbKe-wUzA9qGVw/master.dat] using a single upload]; nested: NotSerializableExceptionWrapper[sdk_client_exception: Failed to connect to service endpoint: ]; nested: IOException[Connect timed out];\nCaused by: java.io.IOException: Unable to upload object [tests-xE9qIVoPSbKe-wUzA9qGVw/master.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:492)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:128)\n\tat java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:37)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:126)\n\tat org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:118)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:263)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1677)\n\tat org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: NotSerializableExceptionWrapper[sdk_client_exception: Failed to connect to service endpoint: ]; nested: IOException[Connect timed out];\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)\n\tat com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)\n\tat com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)\n\tat com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)\n\tat com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)\n\tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)\n\tat com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)\n\tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$19(S3BlobContainer.java:490)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:46)\n\tat java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:45)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:490)\n\t... 13 more\nCaused by: java.io.IOException: Connect timed out\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)\n\tat java.net.Socket.connect(Socket.java:633)\n\tat sun.net.NetworkClient.doConnect(NetworkClient.java:178)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:498)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:603)\n\tat sun.net.www.http.HttpClient.<init>(HttpClient.java:246)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:351)\n\tat sun.net.www.http.HttpClient.New(HttpClient.java:373)\n\tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1287)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)\n\tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057)\n\tat com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)\n\tat 
com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)\n\t... 45 more\n"

why? is setting not effect？

warkolm · April 26, 2022, 4:06am

Welcome to our community!

What service are you using?

forget6 · April 26, 2022, 7:56am

our s3 team reply: no flow to s3 from my IP，i guess the setting maybe not effect，how can i confirm it.[quote="warkolm, post:2, topic:303208, full:true"]
Welcome to our community!

What service are you using?
[/quote]

our company s3, our s3 team reply: no flow from my ip to s3

DavidTurner · April 26, 2022, 8:32am

Caused by: NotSerializableExceptionWrapper[sdk_client_exception: Failed to connect to service endpoint: ]; nested: IOException[Connect timed out];
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)
	at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)
	at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)
	at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
	at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
	at java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)
	at org.elasticsearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:222)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)

This indicates that Elasticsearch is trying (and failing) to get instance credentials from the instance metadata service, which means it does not see the access key and secret key in the keystore. If you configured your keystore after starting Elasticsearch you will need to reload secure settings. Also make sure the keystore is in the correct location.

forget6 · April 26, 2022, 9:04am

i have restarted server.but it still cannot work. my config is a new dir.i use export to redirect it.

ES_DIR=/data/ins_1
ES_HOME=/data/es/elasticsearch
ES_PATH_CONF=${ES_DIR}/config
export ES_PATH_CONF

i cp all config file in this new dir /data/ins_1/config. but i find after i run

elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key

the config file Elasticsearch.keystore is not same as my config dir.

cd /data/ins_1/config
md5sum elasticsearch.keystore 
3b6c1e50190823cf91dca7191293097d  elasticsearch.keystore

cd /data/es/elasticsearch/config
md5sum elasticsearch.keystore 
3e20d606979ac6c17123895b9fae32b0  elasticsearch.keystore

Elasticsearch-keystore add only change file in defalut dir?
the keystore only save in Elasticsearch.keystore?
if i cp the Elasticsearch.keystore to my config dir can solve it?
how can i know which Elasticsearch.keystore is using?

forget6 · April 26, 2022, 9:32am

i solve it. the keystore is wrong. thank you

system · May 24, 2022, 9:32am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.