Is it possible to only restore the incremental portion of snapshot?

[fate9834]

Continuing the discussion from Incremental Snapshot / Restore:

The example from David was:

Snapshot1 pushed segments 1, 2, 3 in repository X
Snapshot2 needs to backup segments 1, 2, 3, 4 (4 contains the new data). But repository X already has segments 1, 2, 3. So the _snapshot action will only copy for snapshot2 the segment 4.

Let's say I have two Elasticsearch clusters, A & B.
At time 0, I created Snapshot1 at A. It contains segment 1, 2, 3.
At time 1, I restored Snapshot1 at B. B now has segment 1, 2, 3.
At time 2, I created the incremental Snapshot2. it contains segment 1, 2, 3, 4.

Now, is it possible that I restore Snapshot2 at B, when B already contains segment 1,2,3?
If yes, will it restore segment 4 only? Or will it destroy all indexes in B first, and then restore segment 1, 2, 3, 4?

Thanks!

[Igor_Motov]

@fate9834

Now, is it possible that I restore Snapshot2 at B, when B already contains segment 1,2,3?

If yes, will it restore segment 4 only? Or will it destroy all indexes in B first, and then restore segment 1, 2, 3, 4?

Yes, in most cases we will only restore segment 4. You need to be careful though, if the primary and replica flip on A for whatever reason, the new primary (former replica) might have a completely different set of segments. In this case we are going to overwrite all files.

[fate9834]

Hi Igor, Thanks for your response! I have two more questions

In documentation of Snapshot & Restore:

"The restore operation can be performed on a functioning cluster. However, an existing index can be only restored if it’s closed and has the same number of shards as the index in the snapshot. The restore operation automatically opens restored indices if they were closed"

My first question is: if an existing index index1 in my cluster B is re-opened by the Restore operation, during the Restore operation is running, is it possible to keep writing new documents into index1?

My second question is: if I'm not adding any new Elasticsearch nodes, will all existing indices be stable? Could you please, briefly, give an example where the shards can flip in that case?

Thanks!

[Igor_Motov]

@fate9834

My first question is: if an existing index index1 in my cluster B is re-opened by the Restore operation, during the Restore operation is running, is it possible to keep writing new documents into index1?

No, during the restore operation the restored index is going to be in the RED state and until shards are fully restored from the snapshot all write and read operations against restored shards are going to fail.

@fate9834

My second question is: if I’m not adding any new Elasticsearch nodes, will all existing indices be stable? Could you please, briefly, give an example where the shards can flip in that case?

A replica shard will become primary if the node with the primary shard dies or disconnects from the cluster.

[fate9834]

Thank you so much for the answer!