How can I restore a snapshot from S3 to a different cluster?

Elastic Stack Elasticsearch
1

I have managed to get a snapshot stored on S3, now trying to get that snapshot onto my elasticsearch deployment using an operator on GKE. Currently I have 1 master data pod elasticsearch-config-es-master-data-1 active.

When trying to set up an S3 repository, below is my PUT request followed by the error I see.

PUT _snapshot/s3_repo
{
  "type": "s3",
  "settings": {
    "bucket": "opensearch-snapshots"
  }
}

ERROR:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[s3_repo] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[s3_repo] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-FtGw-UFkSSC4NN_DcfxsgQ/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "The requested metadata is not found at http://[IP address]/latest/meta-data/iam/security-credentials/"
      }
    }
  },
  "status": 500
}

I'm wondering what I'm missing, any help/guidance is greatly appreciated!

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See What is OpenSearch and the OpenSearch Dashboard? | Elastic for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

Removed #opensearch

4

OpenSearch is a different product from Elasticsearch (with some shared heritage ofc) and their snapshot formats are not compatible. If these snapshots were created with OpenSearch then Elasticsearch isn't going to be able to read them.

This indicates that you haven't configured the access_key and secret_key so Elasticsearch is trying to use the EC2 IMDS to obtain the credentials it needs to access the bucket, but IMDS doesn't have any credentials to offer.

5

Thanks for the response, David :pray:t4:

If these snapshots were created with OpenSearch then Elasticsearch isn't going to be able to read them.

I created the snapshot by running a PUT request to the OpenSearch domain https://[OpenSearch domain name].us-west-2.es.amazonaws.com/_snapshot/test-snapshot-repo/test-snapshot, is this considered as 'a snapshot created with OpenSearch'? Just wanted a clarification.

This indicates that you haven't configured the access_key and secret_key

I've tried multiple ways to configure this

  1. Setting as environment variable in my deployment pod elasticsearch-config-es-master-data-1.
  2. Adding them in elasticsearch keystore
bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key
bin/elasticsearch-keystore add s3.client.default.session_token
  1. On the PUT request (I'm using Postman), I have included those 3 fields in the authorization as well.
    Are these approaches incorrect?
6

Yes.

That won't do anything.

That also won't do anything.

That should work, but it does require that you're using the default S3 client and that bin/elasticsearch-keystore is running in the same environment as you launched ES (in particular, it has the same value for ${ES_PATH_CONF}).

7

Oh no... So, for people (sadly me) who want to migrate their team away from OpenSearch to manage their own using elasticsearch operator, what are my alternatives so that data is preserved? :frowning:

That should work, but it does require that you're using the default S3 client

Is this referring to the repository I've created that is on my destination cluster? How can I check this?

bin/elasticsearch-keystore is running in the same environment as you launched ES

hmm, the operator I'm using deploys these pods:

elastic-operator-klasjhdakslgjh
elasticsearch-config-es-master-data-1
kibana-config-kb-wooshdhf-akdfjg

I've been making configurations in elasticsearch-config-es-master-data-1 since the pod has elasticsearch and all its other commands in the bin/ directory. So, I'm hoping at least this part is correct.
(thanks again for answering my questions)

8

I think you could find a lot of ideas in this blog post:

1 Like
9

The client is the one you choose in the PUT /_snapshot/test-snapshot-repo call. By default it's default. See the docs for more info:

1 Like
10

Thank you for this resource, David. Seems like a lengthy process but at least I have a starting point and can kinda get an idea what I need to do :smiley:

11

Ok, my configuration is correct then! Thanks again for the information and help! :smiley: