Backup and restore a elasticsearch cluster

Elastic Stack Elasticsearch
1

I have a local Elasticsearch cluster consisting of 3 master nodes and 3 data nodes, all the 6 servers are VMs inside Google Cloud. I create a snapshot repository to google GCS bucket, and save snapshots there.

The instructions followed are here: Google Cloud Storage Repository Plugin | Elasticsearch Plugins and Integrations [7.16] | Elastic, and sample Backup and restore ElasticSearch data using GCS | by Chandrapal Badshah | Medium.

The two main issues I found are:

1, only GCP service account's key.json file works for authentication to GCS repository plugin. I can not fall back to use the GCP service account associated with the GCP Virtual Machines. While the link says it can fall back. Getting started | Elasticsearch Plugins and Integrations [7.16] | Elastic
The running Elasticsearch version is 7.8.0.

2, The GCP service account key.json has to be installed into keystore file in all 6 VMs, to get repository successfully set up, why? I installed it onto only one master node, then try repository creation but it failed with 500 return code.

Anyone in this area could shed a light into it, I'm trying to avoid key.json file, as it is rotated monthly for security reasons. Thanks,

Best,

2

What did you try (i.e. what exactly was in your config) and what happened to indicate that it didn't work? I believe as long as the credentials_file setting is absent then this should work.

Note that this is a secure setting so you will need to use elasticsearch-keystore list to ensure that it is removed.

All nodes need direct access to the repository to avoid the dreadful bottleneck (and massive network traffic costs) that would result from needing to send all the data through one privileged node.

This version is past EOL which severely limits our ability to investigate problems. You should upgrade to a supported version ASAP.

3

HI DavidTurner,

The config is that in /etc/Elasticsearch/Elasticsearch.yml file, I have the project_id configured,

...
gcs:
  client:
    default:
      project_id: <gcp_project_id>

# /usr/share/elasticsearch/bin/elasticsearch-keystore list
keystore.seed
#

Should I let credential file points to an empty file instead,

say,

# touch /var/tmp/empty_file.yml
# /usr/share/elasticsearch/bin/elasticsearch-keystore  add-file gcs.client.default.credentials_file /var/tmp/empty_file.yml
#

Thanks,