Cloud-aws plugin of elasticsearch 2.3.5

vamih · November 2, 2017, 2:25pm

Hi guys,

I installed the cloud-aws plugin of elasticsearch version 2.3.5 which will be used to store the snapshot on AWS S3 compatible. During the install everything went nice and smoothly but when I try to create the snapshot I get the following error:
{
"error" : {
"root_cause" : [ {
"type" : "repository_exception",
"reason" : "[my_s3_repository] failed to create repository"
} ],
"type" : "repository_exception",
"reason" : "[my_s3_repository] failed to create repository",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Unknown [repository] type [s3]"
}
},
"status" : 500
}
I configured elasticsearch like below:
cloud:
aws:
protocol: https
access_key: ***********
secret_key: **********
endpoint: sth.com
proxy:
host: "http://192.168.0.5"
port: "3128"
s3:
protocol: http
signer: "AWS2SignerType"
proxy:
host: 192.168.0.5
port: 3128
You need to know that the endpoint requires Signature v2 enabled.
So in this case can you help me troubleshoot the error and confirm that the elasticsearch config is ok?

dadoonet · November 2, 2017, 3:12pm

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

What is the output of GET _cat/plugins?v ?

vamih · November 2, 2017, 3:54pm

the output of get plugins:

curl -XGET 10.0.1.87:9200/_cat/plugins?v
name         component version type url
inlodmedat01 cloud-aws 2.3.5   j

vamih · November 2, 2017, 4:04pm

and also the other parts of code formatted:

error received:

{
"error" : {
"root_cause" : [ {
"type" : "repository_exception",
"reason" : "[my_s3_repository] failed to create repository"
} ],
"type" : "repository_exception",
"reason" : "[my_s3_repository] failed to create repository",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Unknown [repository] type [s3]"
}
},
"status" : 500
}

and the config of elasticsearch:

cloud:
     aws:
     protocol: https
     access_key: ***********
     secret_key: **********
     endpoint: sth.com
     proxy:
     host: "http://192.168.0.5"
     port: "3128"
     s3:
         protocol: http
         signer: "AWS2SignerType"
         proxy:
         host: 192.168.0.5
         port: 3128

dadoonet · November 2, 2017, 4:36pm

What can you see in elasticsearch logs?

What is this AWS2SignerType?

BTW signer is removed in recent elasticsearch version IIRC.
So even if you can make it work, I don't believe it will still be working in the future. See https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking_60_plugins_changes.html#_s3_repository_plugin

vamih · November 3, 2017, 9:22am

Into elasticsearch logs besides the error above I also see some java errors:

repository] type [s3]];
Caused by: RepositoryException[[my_s3_repository] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [s3]];
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:368)
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Unknown [repository] type [s3]
        at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)
        at org.elasticsearch.repositories.RepositoryTypesRegistry.bindType(RepositoryTypesRegistry.java:49)
        at org.elasticsearch.repositories.RepositoryModule.configure(RepositoryModule.java:58)
        at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
        at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
        at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
        at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:157)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)

Regarding the AWS2SignerType it's a parameter required by AWS, to have each http or https query formatted for signature version 2 or version 4 ( in our case the S3 compatible endpoint requires version 2).
But even if I don't set that parameter the error is the same, the plugin isn't able to recognize the "s3" type.

dadoonet · November 3, 2017, 9:46am

And you have only one node?

vamih · November 3, 2017, 10:05am

No, actually I have a cluster composed of 5 data nodes, 3 master nodes and 2 coordinating nodes. But I installed the plugin only on one data node and tried to create a snapshot just for it.

dadoonet · November 3, 2017, 10:32am

You need to install it on every node.

vamih · November 3, 2017, 1:28pm

I install it on every node, but the error is the same.
Is there a specific configuration of this plugin for this version if I need to go through a proxy to reach the endpoint?

dadoonet · November 3, 2017, 1:58pm

What is the full output of:

GET _cat/plugins?v

What are the full logs?

vamih · November 3, 2017, 2:17pm

Here is the curl:

curl -XGET 10.0.1.87:9200/_cat/plugins?v
name         component version type url
inlodmedat01 cloud-aws 2.3.5   j

And the logs:

[2017-11-03 13:11:55,025][WARN ][rest.suppressed          ] path: /_snapshot/my_s3_repository, params: {pretty=, repository=my_s3_repository}
RemoteTransportException[[inlodmemas01][10.0.1.88:9300][cluster:admin/repository/put]]; nested: RepositoryException[[my_s3_repository] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [s3]];
Caused by: RepositoryException[[my_s3_repository] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [s3]];
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:368)
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Unknown [repository] type [s3]
        at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)
        at org.elasticsearch.repositories.RepositoryTypesRegistry.bindType(RepositoryTypesRegistry.java:49)
        at org.elasticsearch.repositories.RepositoryModule.configure(RepositoryModule.java:58)
        at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
        at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
        at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
        at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:157)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)

dadoonet · November 3, 2017, 3:50pm

This:

curl -XGET 10.0.1.87:9200/_cat/plugins?v
name         component version type url
inlodmedat01 cloud-aws 2.3.5   j

Shows that the plugin is not available on every node.
You need to install the plugin on every node AND restart each node.

But as I said at the beginning I think that all those efforts are useless as non AWS S3 repositories are not supported in 6.0.

So you will have to fork the project and make it work for your "whatever-S3-compatible" repository.

system · December 1, 2017, 3:51pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.