S3 Repository Config

Elastic Stack Elasticsearch
1

Hi,
I'm getting an error provisioning an s3 repo on elasticsearch 5.6.3 via repository-s3. Can you assist?

config json:

{
  "type": "s3",
  "settings": {
    "endpoint": "s3-server",
    "bucket": "elasticsearch",
    "client": "lab"
  }
}

The error I get is:

{
  "error": {
    "root_cause": [
      {
        "type": "amazon_s3_exception",
        "reason": "amazon_s3_exception: The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000000000000000e8f7-005afd82b2-c73109-default)"
      }
    ],
    "type": "repository_exception",
    "reason": "[s3-snapshots] failed to create repository",
    "caused_by": {
      "type": "amazon_s3_exception",
      "reason": "amazon_s3_exception: The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000000000000000e8f7-005afd82b2-c73109-default)"
    }
  },
  "status": 500
}

Note that the S3 server is ceph. In elasticsearch 2.x i would specify a signertype of S3SignerType. How do I do this for repository-s3?

Regards,
David

2

I don't think this plugin is compatible with CEPH.

3

The cloud-aws plugin was compatible. I'm surprised there isn't feature parity.

4

It was untested so compatible by luck I'd say.
Best thing to do IMO is to fork the S3 plugin and add whatever is needed to make it work with old S3 API.

5

Can't backward compatibility be added as a feature?

6

We don't want to add a feature we are not testing. May be it will come in the future or someone from the community will build it.

For reference see:

7

Ok. Where can I find guidance on how to get started with this? I've never done it before...

8

So, seeing as I'm running 5.6.3, signer settings won't be an issue for me (yet)?

9

That should work indeed. You should see only some deprecation messages I guess.

10

I've pressed on, am getting different errors now. Config looks like this:

cloud.aws.s3.endpoint: s3-server
cloud.aws.s3.signer: S3SignerType

Repo config json:

{
  "type": "s3",
  "settings": {
    "access_key": "access-key-string",
    "secret_key": "secret-key-string",
    "endpoint": "s3-server",
    "bucket": "elasticsearch",
    "client": "lab",
    "base_path": "snapshots/elasticsearch_lab",
    "protocol": "http",
    "compress": "false",
    "read_timeout": "15s",
    "storage-class": "default",
    "chunk_size": "1gb"
  }
}

And the logged exception:

Caused by: java.lang.IllegalArgumentException: Can not access public com.amazonaws.partitions.model.Partitions(java.lang.String,java.util.List) (from class com.amazonaws.partitions.model.Partitions; failed to set access: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks")
                at com.fasterxml.jackson.databind.util.ClassUtil.checkAndFixAccess(ClassUtil.java:505) ~[?:?]
                at com.fasterxml.jackson.databind.deser.impl.CreatorCollector._fixAccess(CreatorCollector.java:271) ~[?:?]
                at com.fasterxml.jackson.databind.deser.impl.CreatorCollector.verifyNonDup(CreatorCollector.java:318) ~[?:?]
                at com.fasterxml.jackson.databind.deser.impl.CreatorCollector.addPropertyCreator(CreatorCollector.java:183) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addDeserializerConstructors(BasicDeserializerFactory.java:497) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:325) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:258) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:216) ~[?:?]
                at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:358) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:245) ~[?:?]
                at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:143) ~[?:?]
                at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:439) ~[?:?]
                at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:3666) ~[?:?]
                at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3558) ~[?:?]
                at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2648) ~[?:?]
                at com.amazonaws.partitions.PartitionsLoader.loadPartitionFromStream(PartitionsLoader.java:96) ~[?:?]
                at com.amazonaws.partitions.PartitionsLoader.build(PartitionsLoader.java:88) ~[?:?]
                at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30) ~[?:?]
                at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:66) ~[?:?]
                at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:54) ~[?:?]
                at com.amazonaws.regions.RegionUtils.getRegionByEndpoint(RegionUtils.java:123) ~[?:?]
                at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:725) ~[?:?]
                at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:86) ~[?:?]
                at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:308) ~[?:?]
                at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$0(S3RepositoryPlugin.java:82) ~[?:?]
                at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:383) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:356) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.3.jar:5.6.3]
                at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.3.jar:5.6.3]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]
                at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171]
11

Sounds like a Security issue. It needs probably some other permissions than the ones the plugin is using for S3 at it uses probably different classes.

I think you need to modify the plugin-security.policy file.

@tlrx might know more about this.

12

@dawiro: Which JVM are you using exactly? (it's printed out in the elasticsearch logs at startup time)

13

JVM info:

$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
14

@tanguy @dadoonet see jvm info above :slight_smile:

15

Thanks. Does the bucket already exists on Ceph?

16

No it doesn't @tanguy .

17

Can you create it on Ceph and try to register the repository again?

Note that on more recent versions of Elasticsearch, the bucket is not created automatically anymore.

18

@tanguy I'll try that. Why has that been changed? It doesn't make sense.

19

What do you mean by "that"?

20

I mean that it makes sense to me that either:

  • repo creation not depend on the bucket existing
  • the repo creates the bucket when it is created, if the bucket does not exist