S3 repo for non-AWS endpoint - configuration problem (guice error)

Elastic Stack Elasticsearch
1

I'm trying to use ES with elasticsearch-cloud-aws 2.7.1 with non-AWS endpoint and I can't figure out right config (examples can't get me further).

I use have this request:
curl -XPUT 'http://localhost:9200/_snapshot/test_s3repo' -d '{
"type" : "s3",
"settings" : {
"endpoint" : "https://s3-compatible-endpoint", "bucket" : "test_bucket", "access_key" : "...", "secret_key" : "..."
}
}'
but I get error which I understand comes from wrong request. What am I missing?

{"error":"RepositoryException[[test_s3repo] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null), S3 Extended Request ID: null\n at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)\n while locating org.elasticsearch.repositories.s3.S3Repository\n while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: AmazonS3Exception[Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null)]; ","status":500}

2

Sometimes providers claim to be S3 compatible but they are compatible with only the old APIs (V2 vs V3).
IIRC it produces that kind of message.

You can try this setting: https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws-usage.html#cloud-aws-usage-signer

Might help but unsure though.

3

How should this cloud.aws.signer be set? As -D Java parameters? Or passed in JSON
I tested both anyway with both AWS3SignerType and S3SignerType but no success.
Interestingly Ceph is mentioned with these problems which is the S3 provider I call (https://github.com/elastic/elasticsearch-cloud-aws/issues/223)

Is the exact same classes used underneath for non-AWS providers as well?
From stacktrace it looks like pure AWS call:

08:49:05.988 [elasticsearch[The Destroyer][clusterService#updateTask][T#1]] WARN  o.e.repositories - [The Destroyer] failed to create repository [test_s3repo]
org.elasticsearch.repositories.RepositoryException: [test_s3repo] failed to create repository
    at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:414) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:371) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:204) [elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:167) [elasticsearch-1.7.3.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null), S3 Extended Request ID: null
  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.s3.S3Repository
  while locating org.elasticsearch.repositories.Repository

1 error
    at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69) ~[elasticsearch-1.7.3.jar:na]
    at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:407) ~[elasticsearch-1.7.3.jar:na]
    ... 9 common frames omitted
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) ~[aws-java-sdk-core-1.10.12.jar:na]
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.10.12.jar:na]
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) ~[aws-java-sdk-core-1.10.12.jar:na]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) ~[aws-java-sdk-core-1.10.12.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3604) ~[aws-java-sdk-s3-1.10.12.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1050) ~[aws-java-sdk-s3-1.10.12.jar:na]
    at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1013) ~[aws-java-sdk-s3-1.10.12.jar:na]
    at org.elasticsearch.cloud.aws.blobstore.S3BlobStore.<init>(S3BlobStore.java:74) ~[elasticsearch-cloud-aws-2.7.1.jar:na]
    at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:125) ~[elasticsearch-cloud-aws-2.7.1.jar:na]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_66]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_66]