How to refresh/update snapshot repositories?

Elastic Stack Elasticsearch
1

I'm moving data from cluster 2x.A to cluster 5x.B, and have the same S3-based repo registered on both. But any new snapshots taken on A don't show up on B.

I've tried updating B's repo registration with both PUT and PATCH, and neither one updates the list of snapshots. I've even unregistered the repo on B, made sure it was no longer listed as a repo, and reregistered it. But I still only see snapshots from the time I originally registered the repo on B.

The new snapshot files and related index data are definitely on S3. And the new snapshots are visible from cluster A.

Does each cluster maintain an independent cache of snapshots, that only gets updated when certain actions are taken? If so, is there a way to have a cluster refresh this cache?

1 Like
2

I don't think you can/should write with a 2.x cluster on a repository which has been upgraded by a 5.x cluster.

One thing you can try is to delete the repo in 5.x and create it again but I'm not sure it will work.

3

Huh. The docs say: "A snapshot of an index created in 2.x can be restored to 5.x.". Are you saying that the 5x cluster has updated the repo itself somehow?

I did unregister the repo in 5x (with the DELETE verb), and register it again. And it still doesn't show the new 2x snapshots. Even tried giving the repo a different name.

4

A snapshot taken in 2.x can indeed be restored in 5.x However, there are two points:

  1. You should never have two different clusters connected to the same repository where both clusters can write to the repository. This is dangerous and can lead to multiple race conditions, even if they are both 2.x clusters. All but one of the clusters should open the repository in a readonly fashion.

  2. In 5.x, the formats also changed to make repositories more resilient. In particular, we introduced the notion of generational index-N files, as opposed to 2.x which just had one index blob that kept getting overwritten. The index blobs maintain the truth of what are the valid snapshots in the repository. When you connect a 5.x cluster to a repository that was created in 2.x, it will notice that the blobs are in the older 2.x format, so it will "upgrade" them to the index-N version. Thereafter, only 5.x clusters should write to the repository. A 2.x cluster that writes a snapshot will only update the index blob, not the index-N blob, and therefore, the 5.x cluster will have no clue about the new snapshots.

If you want to have the 5.x cluster pick up the new snapshots, do the following:

  1. Unregister the repository from the 5.x cluster
  2. Delete all blobs of the index-N format (i.e. index-1, index-2 etc). But do not delete the index blob (the one without a "-N" at the end).
  3. Delete the index.latest blob
  4. Set the 2.x cluster's repository to readonly
  5. Re-register the repository on the 5.x cluster.

Hope this helps clarifying.

7 Likes
5

Thank you for the detailed answer. That helps a lot!

This advice and background should be called out in the snapshot and restore docs, IMO.

6

You are correct, I will make a note to update the docs with this info

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.