404 - missing repository error when trying to restore snapshot

[Eva]

Hello there

I have an ES cluster with 2 nodes and i'm now switching to AWS ES domain and trying to migrate existing ELK logs to new AWS domain. I have registered a repo and also taken a snapshot of existing ES cluster but when restoring this snapshot to new ES domain i see below error.

{"error":{"root_cause":[{"type":"repository_missing_exception","reason":"[news3repository] missing"}],"type":"repository_missing_exception","reason":"[news3repository] missing"},"status":404}

I use below command -

curl -X POST '<new_ES_domain_Endpoint>/_snapshot/news3repository/<snapshot_name>/_restore'

When i run the below, i see two repos that were created:

curl -X GET ":9200/_snapshot/_all"

{"my_s3_repository":{"type":"s3","settings":{"bucket":"XXXXXX","secret_key":"XXXXXX","region":"XXXX","access_key":"XXXXX"}},"news3repository":{"type":"s3","settings":{"bucket":"XXXXXX","secret_key":"XXXXXX","region":"XXXXXX","access_key":"XXXXXX"}}}

but why is the 404 error coming up when posting to a new ES domain endpoint. Kindly help

Thanks

[dadoonet (David Pilato)]

Are you sure that the command

curl -X GET ":9200/_snapshot/_all"

is run on the new cluster and not the old one?
Did you create the repository in the new cluster?

Also which version are you using? Anything in logs?

[Eva]

Hi David
Thank you for the response.

I created repo on the old cluster [old cluster is on 2 separate EC2 instances], created a repo and took a snapshot. Now i want to switch to new AWS Elasticsearch domain and I am trying to migrate old elk cluster logs to this new AWS ES Domain. For this i'm now trying to use the old cluster snapshot to new AWS Elasticsearch domain.

Hope this is more clear.

And the commands are run on old cluster. how do i migrate them to my new AWS ES domain.

[Eva]

to add to, i'm using version 5.6 for both old cluster and new AWS ES domain

[dadoonet (David Pilato)]

How the new cluster could be aware of the repository if you don't create it?

Create the repo on the new cluster first.

[Eva]

Ok, so i need to create a repo on new cluster and then take a snapshot of old cluster ?

[dadoonet (David Pilato)]

I believe you already created a snapshot of the old cluster so the last step is not needed.

[Eva]

Hi David

I tried couple of times and i get this error when i register the repo on new AWS ES domain:

{"Message":"User: anonymous is not authorized to perform: iam:PassRole on resource: arn:aws:iam::--------------------"}

What is the issue here.

Thanks!

[dadoonet (David Pilato)]

I'm not AWS expert and I'm not sure where this message is coming from and how but I'd say that you need to check your credentials in elasticsearch (did you set up the client as well?) and in AWS?

I'd need to understand exactly what you are doing to assist further.

[Christian_Dahlqvist]

I believe you are using the AWS ES service, is that correct? In that case this may be due to limitations or constraints imposed by their service, so you may want to also raise this with AWS support.

[Eva]

Thank you for the response!

That's right! I'm now using AWS ES domain. This is the new domain that I'm going to configure and I require my old ELK(I have a cluster with 2 nodes run on 2 different EC2 instances) logs to be migrated to this new AWS ES domain. For this, as the procedure says I have registered a repository and taken a snapshot of old ES cluster. Now as David mentioned I'm trying to register the repository on new AWS ES endpoint, where I'm failing with the mentioned error.

[dadoonet (David Pilato)]

Did you look at https://www.elastic.co/cloud and https://aws.amazon.com/marketplace/pp/B01N6YCISK ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas...