Failed to restore an incremental snapshot

[praval_singhal]

I have been trying to restore an incremental snapshot for my ES cluster stored in azure storage account.

I am getting this error

{"error":{"root_cause":[{"type":"snapshot_restore_exception","reason":"[elasticsearch_snapshot:snapshot_2-19-2024] snapshot does not
exist"}],"type":"snapshot_restore_exception","reason":"[elasticsearch_snapshot:snapshot_2-19-2024] snapshot does not exist"},"status":500}

Both the source and destination ES cluster have the same storage account name and key properly configured. Also, Both clusters are running the same ES version 7.17.5.

We have verified all the indexes are in closed state and there are no failed indexes.

We have been successfully able to restore an initial snapshot {snapshot} and also an incremental snapshot {snapshot_1-31-2024} but when we are again trying to restore another incremental snapshot {snapshot_2-19-2024}, we are seeing issues.

I can see my snapshot in the response for list API on the source cluster

[DavidTurner]

I suspect you've not followed these docs:

Clusters should only register a particular snapshot repository bucket once. If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. On other clusters, register the repository as read-only.

[praval_singhal]

@DavidTurner We are doing write operations only from one cluster. Other cluster is just reading the snapshots.

Also, can we update this at this point of time when it is already registered and the cluster is running?

[DavidTurner]

@praval_singhal

We are doing write operations only from one cluster. Other cluster is just reading the snapshots.

That's good, but you must follow the docs and register the repository as read-only too.

@praval_singhal

Also, can we update this at this point of time when it is already registered and the cluster is running?

Yes, if it's registered as read-only.

[praval_singhal]

@DavidTurner Currently for both the cluster, the repository setting is this

{
"elasticsearch_snapshot" : {
"type" : "azure",
"uuid" : "some string",
"settings" : { }
}
}

Can I re-register with read-only setting as true on the restoring cluster?

[DavidTurner]

Yes, remove that repository and add it back with readonly: true in the settings.

[praval_singhal]

Thanks. This worked.