Restore from S3 repository_exception

[twilight (Aatif)]

Hello,

This is about restoring on self-managed from a snapshot created by managed stack.

I have ES and Kibana 7.12 installed on Amazon Linux AMI.
I have a cloud based cluster too which has created a number of snapshots on S3.

I am trying to restore one of the cloud snapshot to my EC2 server using the /_snapshot/.../_restore command in Dev tools, I am facing repository_exception, detailed message below:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[s3-repo] Could not read repository data because the contents of the repository do not match its expected state. This is likely the result of either concurrently modifying the contents of the repository by a process other than this cluster or an issue with the repository's underlying storage. The repository has been disabled to prevent corrupting its contents. To re-enable it and continue using it please remove the repository from the cluster and add it again to make the cluster recover the known state of the repository from its physical contents."
      }
    ],
    "type" : "repository_exception",
    "reason" : "[s3-repo] Could not read repository data because the contents of the repository do not match its expected state. This is likely the result of either concurrently modifying the contents of the repository by a process other than this cluster or an issue with the repository's underlying storage. The repository has been disabled to prevent corrupting its contents. To re-enable it and continue using it please remove the repository from the cluster and add it again to make the cluster recover the known state of the repository from its physical contents."
  },
  "status" : 500
}

Let me add here that I have only one node in EC2 cluster, while the cloud cluster from where these snapshots were created has 2 nodes.

Is it due to the number of nodes or something else?

Thanks

[warkolm (Mark Walkom)]

Did you try doing what it suggested to see if that resolves things?

[stephenb (Stephen Brown)]

That error is typically when you register a repository as read / write on more than one cluster.

Only one cluster can / should register the repo as write, all the other should register the repository as read only.

[twilight (Aatif)]

Thanks, setting as read-only worked.

Also please tell can I just select a latest snapshot and it will restore all the current indices with latest data? or there is a sequence of snapshots to restore?

Does restore only brings the indices and data or also the stack configs etc? because I initiated a restore and after some time my Kibana failed to load, showing some "Service not available" message.
Also elasticsearch:9200 url is not accepting the authentication credentials I had set earlier.

Thanks

[twilight (Aatif)]

Yes, setting the repo read-only helped me in restoring which took some time, but it has some strange effect on my overall ES stack, Kibana is not loading, showing "Kibana server is not ready yet".

[twilight (Aatif)]

I was only able to start Elasticsearch and Kibana services after disabling the xpack auth.
Why would restore affect the user credentials?

[stephenb (Stephen Brown)]

Well if you restored all the system indices that has user, roles etc. so you need to be thoughtful when you restore to consider if you want to do that ... or just restore data etc...

[twilight (Aatif)]

I restored indices while renaming them to avoid the conflict though. And after restore, I also tried setting passwords again using password-setup utility, there was an error about apm_system user while setting password.

How can I skip system or built-in indices to restore on the new cluster? or how can I select what indices to include while creating a snapshot from old cluster?
After all I just need data to be transferred from old to new cluster.

Thanks

[stephenb (Stephen Brown)]

When you restore though the Kibana UI you can select which indices you want to restore and which not. You can use wildcards etc.

You can do that through the API as well but probably easier through Kibana the first time.

Most of this is covered in the docs.

With respect to the password etc I am not sure what state you are in.

[twilight (Aatif)]

Thank you, I have restored selective indices