S3 Repository Config

dawiro · May 17, 2018, 1:38pm

Hi,
I'm getting an error provisioning an s3 repo on elasticsearch 5.6.3 via repository-s3. Can you assist?

config json:

{
  "type": "s3",
  "settings": {
    "endpoint": "s3-server",
    "bucket": "elasticsearch",
    "client": "lab"
  }
}

The error I get is:

{
  "error": {
    "root_cause": [
      {
        "type": "amazon_s3_exception",
        "reason": "amazon_s3_exception: The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000000000000000e8f7-005afd82b2-c73109-default)"
      }
    ],
    "type": "repository_exception",
    "reason": "[s3-snapshots] failed to create repository",
    "caused_by": {
      "type": "amazon_s3_exception",
      "reason": "amazon_s3_exception: The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000000000000000e8f7-005afd82b2-c73109-default)"
    }
  },
  "status": 500
}

Note that the S3 server is ceph. In elasticsearch 2.x i would specify a signertype of S3SignerType. How do I do this for repository-s3?

Regards,
David

dadoonet · May 17, 2018, 2:12pm

I don't think this plugin is compatible with CEPH.

dawiro · May 17, 2018, 2:15pm

The cloud-aws plugin was compatible. I'm surprised there isn't feature parity.

dadoonet · May 17, 2018, 2:31pm

It was untested so compatible by luck I'd say.
Best thing to do IMO is to fork the S3 plugin and add whatever is needed to make it work with old S3 API.

dawiro · May 17, 2018, 2:42pm

Can't backward compatibility be added as a feature?

dadoonet · May 17, 2018, 2:59pm

We don't want to add a feature we are not testing. May be it will come in the future or someone from the community will build it.

For reference see:

dawiro · May 18, 2018, 6:48am

Ok. Where can I find guidance on how to get started with this? I've never done it before...

dawiro · May 18, 2018, 8:08am

So, seeing as I'm running 5.6.3, signer settings won't be an issue for me (yet)?

dadoonet · May 18, 2018, 8:23am

That should work indeed. You should see only some deprecation messages I guess.

dawiro · May 18, 2018, 8:24am

I've pressed on, am getting different errors now. Config looks like this:

cloud.aws.s3.endpoint: s3-server
cloud.aws.s3.signer: S3SignerType

Repo config json:

{
  "type": "s3",
  "settings": {
    "access_key": "access-key-string",
    "secret_key": "secret-key-string",
    "endpoint": "s3-server",
    "bucket": "elasticsearch",
    "client": "lab",
    "base_path": "snapshots/elasticsearch_lab",
    "protocol": "http",
    "compress": "false",
    "read_timeout": "15s",
    "storage-class": "default",
    "chunk_size": "1gb"
  }
}

And the logged exception:

Caused by: java.lang.IllegalArgumentException: Can not access public com.amazonaws.partitions.model.Partitions(java.lang.String,java.util.List) (from class com.amazonaws.partitions.model.Partitions; failed to set access: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks")
                at com.fasterxml.jackson.databind.util.ClassUtil.checkAndFixAccess(ClassUtil.java:505) ~[?:?]
                at com.fasterxml.jackson.databind.deser.impl.CreatorCollector._fixAccess(CreatorCollector.java:271) ~[?:?]
                at com.fasterxml.jackson.databind.deser.impl.CreatorCollector.verifyNonDup(CreatorCollector.java:318) ~[?:?]
                at com.fasterxml.jackson.databind.deser.impl.CreatorCollector.addPropertyCreator(CreatorCollector.java:183) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addDeserializerConstructors(BasicDeserializerFactory.java:497) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:325) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:258) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:216) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:358) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:245) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:143) ~[?:?]
                at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:439) ~[?:?]
                at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:3666) ~[?:?]
                at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3558) ~[?:?]
                at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2648) ~[?:?]
                at com.amazonaws.partitions.PartitionsLoader.loadPartitionFromStream(PartitionsLoader.java:96) ~[?:?]
                at com.amazonaws.partitions.PartitionsLoader.build(PartitionsLoader.java:88) ~[?:?]
                at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30) ~[?:?]
                at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:66) ~[?:?]
                at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:54) ~[?:?]
                at com.amazonaws.regions.RegionUtils.getRegionByEndpoint(RegionUtils.java:123) ~[?:?]
                at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:725) ~[?:?]
                at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:86) ~[?:?]
                at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:308) ~[?:?]
                at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$0(S3RepositoryPlugin.java:82) ~[?:?]
                at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:383) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:356) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.3.jar:5.6.3]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]
                at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171]

dadoonet · May 18, 2018, 8:45am

Sounds like a Security issue. It needs probably some other permissions than the ones the plugin is using for S3 at it uses probably different classes.

I think you need to modify the plugin-security.policy file.

@tlrx might know more about this.

tanguy · May 18, 2018, 9:11am

@dawiro: Which JVM are you using exactly? (it's printed out in the elasticsearch logs at startup time)

dawiro · May 18, 2018, 9:38am

JVM info:

$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

dawiro · May 18, 2018, 12:07pm

@tanguy @dadoonet see jvm info above

tanguy · May 18, 2018, 12:12pm

Thanks. Does the bucket already exists on Ceph?

dawiro · May 18, 2018, 12:13pm

No it doesn't @tanguy .

tanguy · May 18, 2018, 12:58pm

Can you create it on Ceph and try to register the repository again?

Note that on more recent versions of Elasticsearch, the bucket is not created automatically anymore.

dawiro · May 18, 2018, 2:23pm

@tanguy I'll try that. Why has that been changed? It doesn't make sense.

tanguy · May 18, 2018, 2:40pm

What do you mean by "that"?

dawiro · May 25, 2018, 12:06pm

I mean that it makes sense to me that either:

repo creation not depend on the bucket existing
the repo creates the bucket when it is created, if the bucket does not exist

Topic		Replies	Views
Elasticsearch ceph s3 repository 400; "Error Code: InvalidArgument; Request ID: null" Elasticsearch	2	1263	March 29, 2017
S3_repository error Elasticsearch	5	3194	October 21, 2020
Repository-s3 plugin does not support S3-compatible services as documentation says Elasticsearch snapshot-and-restore	3	679	March 10, 2021
I cant access S3 repo CEPH using repository-s3 Elasticsearch	2	1367	June 8, 2019
Cloud-aws plugin of elasticsearch 2.3.5 Elasticsearch	13	3296	December 1, 2017

S3 Repository Config

Related topics