TransportClient and creating an S3 snapshot repository

Elastic Stack Elasticsearch
1

es version: 2.3.3

I'm currently attempting to create an s3 snapshot repository using the java TransportClient. I'm getting the following error:

RepositoryException[[028181cde20d44a8bb3009f71daf8524_backup] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [S3]];
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
[snip]
Caused by: java.lang.IllegalArgumentException: Unknown [repository] type [S3]
at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)

I tried adding the cloud aws plugin to the transport client like this:

new TransportClient
                .Builder()
                .settings(Settings.builder().put("cluster.name", "cluster_name"))
                .addPlugin(CloudAwsPlugin.class)
                .build()

but I'm still getting the same error.

EDIT: Figured it out. needed to enable the plugin via the cloud.enabled setting

new TransportClient
                .Builder()
                .settings(Settings.builder().put("cluster.name", config.clusterName).put("cloud.enabled", true))
                .addPlugin(CloudAwsPlugin.class)
                .build()

also needed to set the type of my repo to "s3" instead of "S3". The capital S worked in es 1.7.4, but not in 2.3.3.

2

Thank you for this thread. It's super interesting.

I opened https://github.com/elastic/elasticsearch/issues/19349

BTW, instead of editing your question, it's better to add an answer so anyone will be aware of the fix and we can then mark the answer as solving the problem.

3

BTW, instead of editing your question, it's better to add an answer so anyone will be aware of the fix and we can then mark the answer as solving the problem.

Good to know! Copied and pasted the answer below in order to mark as solved:

Needed to enable the plugin via the cloud.enabled setting

new TransportClient
                .Builder()
                .settings(Settings.builder().put("cluster.name", config.clusterName).put("cloud.enabled", true))
                .addPlugin(CloudAwsPlugin.class)
                .build()

also needed to set the type of my repo to "s3" instead of "S3". The capital S worked in es 1.7.4, but not in 2.3.3.

4

So that looks strange. I tried to look at the code today and reproduce the issue you mentioned and was not able.

Here is what I did:

  • Launch a node with default settings and cloud-aws plugin.
  • Create a TransportClient like this:
        TransportClient client = new TransportClient.Builder()
                .addPlugin(CloudAwsPlugin.class)
                .build();
        client.addTransportAddress(new InetSocketTransportAddress(new InetSocketAddress("127.0.0.1", 9300)));

        PutRepositoryResponse putRepositoryResponse = client.admin().cluster().preparePutRepository("test-repo")
                .setType("s3").setSettings(Settings.EMPTY).get();

This does not raise the issue you mentioned.
Also in code we default cloud.enabled to true. So I can't understand how this happened.

Could you share more details?

5

Thinking about his more, I think the actual error was s3 vs S3. Under 1.7 we had a script to create repos with type "S3" which worked under 1.7, but I think elasticsearch only accepts "s3" as a type now. The cloud.enabled setting was a red herring.

Oh, one other thing re: s3 vs S3, I was wondering if it was intentional behaviour to only accept "s3". It seems like a toLowerCase() on this line could allow both: https://github.com/elastic/elasticsearch/blob/dea00a0b16f516e6ffb41c4e8a10b24dd992b16e/core/src/main/java/org/elasticsearch/repositories/RepositoriesService.java#L375 Alternately it might make sense to call toLowerCase on type in the RepositoryMetaData constructor instead? https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/cluster/metadata/RepositoryMetaData.java#L44

6

Good to know.

I'm wondering what would happen if you create a repository in an elasticsearch cluster with S3 as a type in a 1.7 cluster, then upgrade the cluster to 2.3.

Do you think you could test such a scenario?

If it does not work, I'm unsure if the upgrade plugin is able to catch that.

7

Forget my comment as I saw you wrote S3 snapshot repositories not working after upgrade from 1.7 to 2.3