How to reliably verify a snapshot restore succeeded? Failed shards disappear from _recovery

Elastic Stack Elasticsearch
1

Hello,

I'd like to raise a possible problem I recently ran into in an internal application, and which I see the current Curator implementation mirrors as well (curator/curator/utils.py at v5.8.4 · elastic/curator · GitHub, lines 1744–1756).

Although I've read the documentation (Restore a snapshot | Elasticsearch Guide [7.17] | Elastic), it's not clear to me how exactly you can make sure a RESTORE has completed successfully.
I assume a combination of REST requests is needed, and I'd like to confirm that my understanding is correct.

From what I understand, the following mechanism would be required.

GET my-index/_recovery
=> this returns stages such as INIT/INDEX/VERIFY_INDEX/TRANSLOG/FINALIZE/DONE, but there is no FAILED value.

The trap: if all the shards in the response show DONE, that does not automatically mean the restore finished successfully. Why? Because shards that fail disappear from the response (the shard is deallocated — UNASSIGNED/ALLOCATION_FAILED — and drops out of the /_recovery response entirely). So if a shard failed, you will not get that information from this endpoint. It will tell you that all the remaining shards are DONE, but that does not mean the restore succeeded.

From what I've read in the documentation, a possible and correct solution would be to continue the verification:

GET _cluster/health
If we get GREEN or YELLOW after _recovery has shown the snapshot recoveries completed, then all primary shards are allocated and the restore is complete.
If the status is RED, there are 2 cases:

  1. the error is recoverable
  2. the error is not recoverable
    To find out which case we are in, we need to call GET _cluster/allocation/explain and interpret the data there.

However, both our internal implementation and the one in Curator assume that if all stages are "DONE" in the GET /_recovery response, the restore finished successfully.
This is not true when dealing with errors such as IOException: No space left on device.

Versions: Elasticsearch 7.17.27, RHEL 9

2

Yes this isn't valid. You need to check GET _cluster/health for unassigned shards too. Note that this will also show shards that are unassigned for other reasons.

This version's really old, and no longer maintained or supported. Please upgrade to a supported version ASAP.

3

Thank you, that confirms it. One follow-up to make sure our fix is right: when health is red after the snapshot recoveries look finished, we plan to distinguish "still being retried" from "permanently failed" via GET _cluster/allocation/explain on the unassigned primary — treating can_allocate: no as the terminal-failure signal (and reading the cause from unassigned_info.details). Is that the right criterion, or is there a better one?

On the version: understood, we'll raise the upgrade internally.

4

I don't know of a robust way to distinguish permanent from temporary failures that's accurate in all cases, but what you propose might well work for you. Bear in mind that unassigned_info describes why the shard last became unassigned, but that doesn't necessarily tell you anything about why it cannot become assigned again.

5

Thank you for your answers!

I am thinking of this approach, based on your responses and the documentation.

A poll checks every 5 seconds whether the restore is done:

GET /{restored-indices}/_recovery
├─ at least one shard is not DONE ............... IN_PROGRESS, keep polling
└─ no shards present, or all shards are DONE:
   │
   GET /_cluster/health/{restored-indices}
   ├─ GREEN/YELLOW, but no snapshot recovery activity
   │  observed yet for the current snapshot ..... keep polling
   ├─ GREEN/YELLOW, and snapshot recovery activity
   │  for the current snapshot was observed ..... SUCCESS
   └─ RED:
      │
      GET /_cat/shards/{restored-indices}?format=json&h=index,shard,prirep,state
      (primaries only: prirep=p, state=UNASSIGNED)
      │
      for each UNASSIGNED primary:
         GET /_cluster/allocation/explain (index, shard, primary=true)
         ├─ max_retry decider = NO ............... FAILED (ES stopped retrying)
         ├─ can_allocate = no_valid_shard_copy ... FAILED (for this restore)
         ├─ disk_threshold decider = NO .......... FAILED (our app cannot free disk — policy)
         └─ anything else (transient) ............ keep polling

plus a bounded limit on how long the "red with no verdict" state may
persist (n configurable minutes).

One last factual question: is the max_retry decider returning NO ("shard has
exceeded the maximum number of retries") the correct signal that ES has stopped
automatic allocation attempts for the current retry cycle, until the retry
counter is explicitly reset with POST /_cluster/reroute?retry_failed=true?

6

Normally I would expect to just GET _cluster/health with some of its wait_for options, e.g. wait_for_no_initializing_shards and wait_for_status rather than polling the recovery API.

If the max_retry decider returns NO then the retry counter needs to be reset before it'll try again, but IIRC that happens automatically under some circumstances. Still, yes you can manually reset this counter with POST /_cluster/reroute?retry_failed

7

So, just to make it clear for me, you would suggest to only use the health endpoint?

Something like:

GET /_cluster/health/{restored-indices}
    ?wait_for_status=yellow
    &wait_for_no_initializing_shards=true
    &timeout=30s

So that the logic becomes:

restore started
      │
      ▼
repeat until deadline
      │
      ├── health reaches YELLOW/GREEN
      │       └── SUCCESS
      │
      └── timeout (30s)
              │
              └── try again

My question here would be: how do I choose a valid overall waiting time, and how do I
know the restore cannot progress without intervention, so that I stop waiting early
instead of spinning until the deadline?

Or would you still combine this loop with the allocation/explain check from my
earlier post (max_retry decider = NO) on the red path, so known-permanent failures
stop the wait immediately?

I try to understand what approach you would suggest.

Thank you for your answers!

8

To be honest there's no great solution to this level of detailed monitoring. Long-polling the health API in a loop (you can use a longer timeout than 30s) will tell you whether the restore is complete or not but doesn't show much progress information. The recovery API exposes shard-by-shard information about ongoing restore progress but doesn't show all shards.

9

Thanks for your time!

Unfortunately, I'm still not sure how to handle my case.

We call the restore request with wait_for_completion = false. I'd like to be able to determine whether the restore failed or succeeded. If the health request returns GREEN, then I know it succeeded. YELLOW means that it succeeded or that it is not yet decided? I assume it means that it succeeded.

The problem arises when the response is RED. That can mean it failed, but can it also mean it may still succeed later, without manual intervention?

In that case I can poll the health endpoint at some interval, but how do I know whether RED means it has failed or that it is still trying? We have an 80 GB database, so the process may well take a while. Is the only option to assume some maximum time?

The trouble is that if I pick a timeout, I will tell the user the restore failed while the process is in fact still running in the background and might eventually succeed. That is why I'm not sure how to make certain the process has finished 100%, whether the outcome is success or failure.

10

I smell over engineering here. You are trying for 100%. Accept less. Because even IF you convince yourself you’ve reached 100%, you won’t have. I can’t see how you can validate either, how are you simulating all possible failure scenarios?

Your cluster can certainly be red independently of a running/completed restore process. So, at least IMHO, you are mixing concepts.

11

I'm not sure you need to distinguish these cases. Even if it eventually finds a way to recover (e.g. you add a new node to the cluster) the fact that it failed even once is already notable and worthy of further investigation.