How to reliably verify a snapshot restore succeeded? Failed shards disappear from _recovery

Elastic Stack Elasticsearch
1

Hello,

I'd like to raise a possible problem I recently ran into in an internal application, and which I see the current Curator implementation mirrors as well (curator/curator/utils.py at v5.8.4 · elastic/curator · GitHub, lines 1744–1756).

Although I've read the documentation (Restore a snapshot | Elasticsearch Guide [7.17] | Elastic), it's not clear to me how exactly you can make sure a RESTORE has completed successfully.
I assume a combination of REST requests is needed, and I'd like to confirm that my understanding is correct.

From what I understand, the following mechanism would be required.

GET my-index/_recovery
=> this returns stages such as INIT/INDEX/VERIFY_INDEX/TRANSLOG/FINALIZE/DONE, but there is no FAILED value.

The trap: if all the shards in the response show DONE, that does not automatically mean the restore finished successfully. Why? Because shards that fail disappear from the response (the shard is deallocated — UNASSIGNED/ALLOCATION_FAILED — and drops out of the /_recovery response entirely). So if a shard failed, you will not get that information from this endpoint. It will tell you that all the remaining shards are DONE, but that does not mean the restore succeeded.

From what I've read in the documentation, a possible and correct solution would be to continue the verification:

GET _cluster/health
If we get GREEN or YELLOW after _recovery has shown the snapshot recoveries completed, then all primary shards are allocated and the restore is complete.
If the status is RED, there are 2 cases:

  1. the error is recoverable
  2. the error is not recoverable
    To find out which case we are in, we need to call GET _cluster/allocation/explain and interpret the data there.

However, both our internal implementation and the one in Curator assume that if all stages are "DONE" in the GET /_recovery response, the restore finished successfully.
This is not true when dealing with errors such as IOException: No space left on device.

Versions: Elasticsearch 7.17.27, RHEL 9

2

Yes this isn't valid. You need to check GET _cluster/health for unassigned shards too. Note that this will also show shards that are unassigned for other reasons.

This version's really old, and no longer maintained or supported. Please upgrade to a supported version ASAP.

3

Thank you, that confirms it. One follow-up to make sure our fix is right: when health is red after the snapshot recoveries look finished, we plan to distinguish "still being retried" from "permanently failed" via GET _cluster/allocation/explain on the unassigned primary — treating can_allocate: no as the terminal-failure signal (and reading the cause from unassigned_info.details). Is that the right criterion, or is there a better one?

On the version: understood, we'll raise the upgrade internally.

4

I don't know of a robust way to distinguish permanent from temporary failures that's accurate in all cases, but what you propose might well work for you. Bear in mind that unassigned_info describes why the shard last became unassigned, but that doesn't necessarily tell you anything about why it cannot become assigned again.

5

Thank you for your answers!

I am thinking of this approach, based on your responses and the documentation.

A poll checks every 5 seconds whether the restore is done:

GET /{restored-indices}/_recovery
├─ at least one shard is not DONE ............... IN_PROGRESS, keep polling
└─ no shards present, or all shards are DONE:
   │
   GET /_cluster/health/{restored-indices}
   ├─ GREEN/YELLOW, but no snapshot recovery activity
   │  observed yet for the current snapshot ..... keep polling
   ├─ GREEN/YELLOW, and snapshot recovery activity
   │  for the current snapshot was observed ..... SUCCESS
   └─ RED:
      │
      GET /_cat/shards/{restored-indices}?format=json&h=index,shard,prirep,state
      (primaries only: prirep=p, state=UNASSIGNED)
      │
      for each UNASSIGNED primary:
         GET /_cluster/allocation/explain (index, shard, primary=true)
         ├─ max_retry decider = NO ............... FAILED (ES stopped retrying)
         ├─ can_allocate = no_valid_shard_copy ... FAILED (for this restore)
         ├─ disk_threshold decider = NO .......... FAILED (our app cannot free disk — policy)
         └─ anything else (transient) ............ keep polling

plus a bounded limit on how long the "red with no verdict" state may
persist (n configurable minutes).

One last factual question: is the max_retry decider returning NO ("shard has
exceeded the maximum number of retries") the correct signal that ES has stopped
automatic allocation attempts for the current retry cycle, until the retry
counter is explicitly reset with POST /_cluster/reroute?retry_failed=true?

6

Normally I would expect to just GET _cluster/health with some of its wait_for options, e.g. wait_for_no_initializing_shards and wait_for_status rather than polling the recovery API.

If the max_retry decider returns NO then the retry counter needs to be reset before it'll try again, but IIRC that happens automatically under some circumstances. Still, yes you can manually reset this counter with POST /_cluster/reroute?retry_failed