Detected a corrupted repository , index<name> references an unknown snapshot uuid

[spreet (Preet)]

Hi, All,

We are trying to migrate a self-managed elastic search cluster (version 5.3) to a different Elasticsearch cluster with version 7.1. Below steps works for the first time but fails when I try to restore another snapshot from 5.3 to 7.1:

First attempt

1. Take a snapshot(snap_1) from 5.3 to a S3 repo (snapshot_repo)
2. Spin up a temp cluster with version 6.8 as a direct restore from 5.3 to 7.1 is not allowed.
3. Register snapshot_repo for 6.8 (same repo as 5.3)
4. Restore snap_1 on to 6.8
5. Reindex data (To make mapping compatible with 6.8)
6. Take a new snapshot on 6.8 (snap_2)
7. Register snapshot_repo for 7.1 (same repo as 5.3)
8. Restore snap_2 on 7.1
9. Delete 6.8 version ES cluster

At this time everything works as expected. The number of indices, documents, mapping, etc are the same as the 5.3 version. After this, I tried to restore a new snapshot from 5.3

Second attempt:

1. Take a snapshot(snap_3) from 5.3 to a S3 repo (snapshot_repo)
2. Spin up a temp cluster with version 6.8 as a direct restore from 5.3 to 7.1 is not allowed.
3. Register snapshot_repo for 6.8 (same repo as 5.3)
4. Restore snap_3 on to 6.8 => Failed (see below)

org.elasticsearch.ElasticsearchParseException: Detected a corrupted repository, index [__PATH__] references an unknown snapshot uuid [6YLEH-C_RM6SA6TPYpI9CQ]
	at org.elasticsearch.repositories.RepositoryData.snapshotsFromXContent(RepositoryData.java:466) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:800) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.snapshots.RestoreService.restoreSnapshot(RestoreService.java:184) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:77) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:40) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:124) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:211) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:760) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.0.jar:6.8.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

Any ideas? What can I do to avoid this?

[DavidTurner]

I suspect that once you've written to a repository with version X you probably can't write to it with a version before X without damaging it. I would recommend using two different repositories for the two steps.

[spreet (Preet)]

Thanks, David.

Do you mean I should take snapshot from 6.8 version to a different repository? I will have to use same repo as 5.3 to restore it to 6.8. Right?

Also, at this time I am not even able to get list of snapshots from 6.8:

GET _snapshot/es_s3_snapshot_repository/_all?pretty

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "Detected a corrupted repository, index [<Index name>] references an unknown snapshot uuid [keU32XA2TNyyei949lVHIQ]"
      }
    ],
    "type" : "parse_exception",
    "reason" : "Detected a corrupted repository, index [<index name>] references an unknown snapshot uuid [keU32XA2TNyyei949lVHIQ]"
  },
  "status" : 400
}

I can see list of snapshots from 5.3 cluster but not with 6.8. How do I recover from this?

I can try something like this :

1. Register a new repo for 5.3 (repo2)
2. Take a snapshot from 5.3 (repo2_snap_1)
3. Spin up 6.8 cluster
4. Register repo2 for 6.8
   4 Restore repo2_snap_1 on 6.8
5. Reindex data on 6.8
6. Register a new repo on 6.8 (repo3)
7. Take a new snapshot(repo3_snap1) from 6.8
8. Register repo3 on 7.1 version
9. Restore repo3_snap1 on 7.1
   10 Delete 6.8 cluster

[DavidTurner]

@spreet

I can try something like this :

Yes, I think that sounds good.

[spreet (Preet)]

David, Thanks for the pointer. Using different repo's for each ES version fixed this issue. But I still don't know how to fix previous repo with mixed snapshots from 2 different versions of ES

[DavidTurner]

I don't think there's a way to rescue a repository that's in that state, sorry.