ES 6.4 s3 feature with endpoint as hostname

Elastic Stack Elasticsearch
1

Hello Team,

I am using ES 6.4 version along with S3 plugin v 6.4.3 installed in the cluster, and have configured end point to be minio service host name which is internal to the cluster.

Configuration:

 s3.client.default.endpoint: "http://minio-service:9000" 
 s3.client.default.protocol: http

API request:

curl -X PUT "elastic:9200/_snapshot/myminio" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "elasticsearch",
    "endpoint": "http://minio-service:9000",
    "protocol": "http"
  }
}
'

With the above set up when I run the snapshot call I get below error:

{
   "error":{
      "root_cause":[
         {
            "type":"sdk_client_exception",
            "reason":"sdk_client_exception: Unable to execute HTTP request: elasticsearch.minio-service"
         }
      ],
      "type":"repository_exception",
      "reason":"[myminio] cannot create blob store",
      "caused_by":{
         "type":"sdk_client_exception",
         "reason":"sdk_client_exception: Unable to execute HTTP request: elasticsearch.minio-service",
         "caused_by":{
            "type":"i_o_exception",
            "reason":"elasticsearch.minio-service"
         }
      }
   },
   "status":500
}

I feel this is due to DNS style of http request instead of path sytle.

Note: The above setting does work for IP address to be the end point

Can you please guide me here?

Thanks
Sri

2

@ikakavas Can you please help me here ?

Thanks
Sri

3

Please refrain from pinging/addressing people directly. This is a public forum and questions are answered by whoever has the time to put in the effort to do so. We monitor these forums and will provide answers to the best of our ability and availability.

4

Sorry for that, I will wait for someone to have a look at my issue.

5

Please take the time to format your posts using the </> button or triple backticks (```) . This will make it much more probable that someone will take the time to look into your issue and attempt to help you out.

Please, always provide the exact and complete extract from the elasticsearch logs and not a selected part of it

Is that an appropriate workaround for you or do you need to use an FQDN ?

Is the FQDN minio.service resolvable in your system ? i.e. have you configured something to point minio.service to 127.0.0.1 ?

See also : Client Settings | Elasticsearch Plugins and Integrations [7.3] | Elastic

6

and the logs says

Caused by: org.elasticsearch.repositories.RepositoryException: [myminio] cannot create blob store
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:336) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.repositories.s3.S3Repository.blobStore(S3Repository.java:260) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:635) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:218) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.4.3.jar:6.4.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Unable to execute HTTP request: elasticsearch.minio-service
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:74) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_212]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:73) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.createBlobStore(S3Repository.java:253) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.createBlobStore(S3Repository.java:52) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:332) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.repositories.s3.S3Repository.blobStore(S3Repository.java:260) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:635) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:218) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.4.3.jar:6.4.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
Caused by: java.io.IOException: elasticsearch.minio-service
	at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_212]
	at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_212]
	at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_212]
	at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) ~[?:?]
	at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) ~[?:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111) ~[?:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[?:?]
	at com.amazonaws.http.conn.$Proxy25.connect(Unknown Source) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[?:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[?:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[?:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[?:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[?:?]
	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]
	at 
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:74) ~[?:?]
	at

-Sri

7

Also the DNS resolution is working to right IP address.

Name:      minio-service
Address 1: 10.106.65.22 minio-service.test.svc.cluster.local

Here ES is trying to send request as "{bucket}.hostname", in my case I will be needing to work this one off of FQDN but not IP address.

Please let me know if you have any suggestions for me.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.