Elasticsearch backup GCP plugin

Elastic Stack Elasticsearch
1

I'm having trouble securing a backup for elasticsearch running on GCP, kubernetes.
I've tried two approaches.

The first, that seems like the best is taking snapshots and store in a bucket.

I’ve installed this plugin on every pod:

https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-gce.html

Following these instructions:

https://github.com/elastic/elasticsearch/blob/eb8f3462042693b950d8200a4d740f07641b7be4/docs/plugins/repository-gcs.asciidoc#client-settings

I’ve re-started the pods, saved the credentials in elasticsearch keystore, the cluster has full storage permissions.

When I try to configure a repository through the snapshot-API in Elasticsearch I get errors.
"[backup] repository type [gcs] does not exist"

The second approach for backup I’ve tried is :

It works great, but I’m not able to dynamically provision the backup-disk. So I would have to know in advance the disk-space I would need. Any clue of how to be able to set this dynamically?

2

I think this indicates that the plugin is not installed, or at least is not installed everywhere. Can you share the output of GET /_nodes/plugins?filter_path=nodes.*.plugins,nodes.*.name.

3

Thanks for the quick reply!
You're right,
The output is :
{
"nodes": {
"rqwebKVmQKSDmunhEdSHoA": {
"plugins":
},
"XiKeE5UZTnGp90Ob5nKAew": {
"plugins":
}
}
}

But if I go to: /usr/share/elasticsearch/plugins# , I see discovery-gce.
I'm using :
kubectl exec -it [nameOfpod] -- /bin/bash
to access my pod. Is that the wrong approach? I tried SSH into the node, but it wont let me.

4

Are there any messages indicating problems loading plugins in the Elasticsearch server logs?

Is /usr/share/elasticsearch/plugins the right place for plugins? Where is your elasticsearch.yml file?

5

I'm not sure if I'm installing the plugin the right way. The documentation says:
The plugin must be installed on every node in the cluster.
When I SSH into the nodes, it wont let me install the plugin.
I only get: bin/elasticsearch-plugin: No such file or directory
But when I SSH into the pod, I can install the plugin. I get this:

-> Downloading discovery-gce from elastic 
[=================================================] 100%?? 

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission setFactory
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permission

Any clue on which approach I should take?

6

This looks like it's working (presuming there's more text after this too). One possibility is that the installation is not persistent: it's installing the plugin onto the ephemeral storage within the container so that it's not there the next time it starts up.

Are you running this from wherever Elasticsearch is installed? Is Elasticsearch even installed on the nodes or is it all within the containers?

7

I think I have elasticsearch installed on the pods. Not on the nodes, in a K8 cluster. I've been following this guide:

8

Hmm, ok, I'm not familiar with these packages. It does seem pretty strange that something from GoogleCloudPlatform doesn't include the repository-gcs plugin :slight_smile: Perhaps you could open an issue against that repo asking for this to be included?

I am a little confused about the plugin that you're installing. Your original post was talking about backups, i.e., repository-gcs, but in a later post you seem to be installing a discovery plugin, i.e. discovery-gce. I get these confused all the time.

Could you use the official Elasticsearch Helm chart instead? The README doc there includes instructions for installing plugins.

9

Thanks.
I think I had myself confused as well🙂. It's the repository-gcs backup plugin I wanted to install.
I will try with the Helm chart.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.