# How to reliably verify a snapshot restore succeeded? Failed shards disappear from \_recovery **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [August 25, 2026, 1:48pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912 "2026-08-25T13:48:25Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ciprianamza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ciprianamza/32/148115_2.png) [@ciprianamza](https://discuss.elastic.co/u/ciprianamza)\ **Post date:** [August 25, 2026, 1:48pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/1 "2026-08-25T13:48:25Z")
Hello, I'd like to raise a possible problem I recently ran into in an internal application, and which I see the current Curator implementation mirrors as well ([curator/curator/utils.py at v5.8.4 · elastic/curator · GitHub](https://github.com/elastic/curator/blob/v5.8.4/curator/utils.py), lines 1744–1756). Although I've read the documentation ([Restore a snapshot | Elasticsearch Guide [7.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/snapshots-restore-snapshot.html)), it's not clear to me how exactly you can make sure a RESTORE has completed successfully. I assume a combination of REST requests is needed, and I'd like to confirm that my understanding is correct. From what I understand, the following mechanism would be required. `GET my-index/_recovery` =\> this returns stages such as INIT/INDEX/VERIFY\_INDEX/TRANSLOG/FINALIZE/DONE, but there is no FAILED value. The trap: if all the shards in the response show DONE, that does not automatically mean the restore finished successfully. Why? Because shards that fail disappear from the response (the shard is deallocated — UNASSIGNED/ALLOCATION\_FAILED — and drops out of the /\_recovery response entirely). So if a shard failed, you will not get that information from this endpoint. It will tell you that all the remaining shards are DONE, but that does not mean the restore succeeded. From what I've read in the documentation, a possible and correct solution would be to continue the verification: `GET _cluster/health` If we get GREEN or YELLOW after \_recovery has shown the snapshot recoveries completed, then all primary shards are allocated and the restore is complete. If the status is RED, there are 2 cases: 1. the error is recoverable 2. the error is not recoverable To find out which case we are in, we need to call `GET _cluster/allocation/explain` and interpret the data there. However, both our internal implementation and the one in Curator assume that if all stages are "DONE" in the `GET /_recovery` response, the restore finished successfully. This is not true when dealing with errors such as `IOException: No space left on device`. Versions: Elasticsearch 7.17.27, RHEL 9 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 25, 2026, 2:37pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/2 "2026-08-25T14:37:59Z")
> [@ciprianamza](#): > > However, both our internal implementation and the one in Curator assume that if all stages are "DONE" in the `GET /_recovery` response, the restore finished successfully. Yes this isn't valid. You need to check `GET _cluster/health` for unassigned shards too. Note that this will also show shards that are unassigned for other reasons. > [@ciprianamza](#): > > Versions: Elasticsearch 7.17.27 This version's really old, and no longer maintained or supported. Please upgrade to a supported version ASAP. ---
**Author:** ![ciprianamza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ciprianamza/32/148115_2.png) [@ciprianamza](https://discuss.elastic.co/u/ciprianamza)\ **Post date:** [August 25, 2026, 2:45pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/3 "2026-08-25T14:45:10Z")
Thank you, that confirms it. One follow-up to make sure our fix is right: when health is red after the snapshot recoveries look finished, we plan to distinguish "still being retried" from "permanently failed" via `GET _cluster/allocation/explain` on the unassigned primary — treating `can_allocate: no` as the terminal-failure signal (and reading the cause from `unassigned_info.details`). Is that the right criterion, or is there a better one? On the version: understood, we'll raise the upgrade internally. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 25, 2026, 3:02pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/4 "2026-08-25T15:02:01Z")
I don't know of a robust way to distinguish permanent from temporary failures that's accurate in all cases, but what you propose might well work for you. Bear in mind that `unassigned_info` describes why the shard last became unassigned, but that doesn't necessarily tell you anything about why it cannot become assigned again. ---
**Author:** ![ciprianamza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ciprianamza/32/148115_2.png) [@ciprianamza](https://discuss.elastic.co/u/ciprianamza)\ **Post date:** [August 26, 2026, 7:54am UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/5 "2026-08-26T07:54:35Z")
Thank you for your answers! I am thinking of this approach, based on your responses and the documentation. A poll checks every 5 seconds whether the restore is done: ```auto GET /{restored-indices}/_recovery ├─ at least one shard is not DONE ............... IN_PROGRESS, keep polling └─ no shards present, or all shards are DONE: │ GET /_cluster/health/{restored-indices} ├─ GREEN/YELLOW, but no snapshot recovery activity │ observed yet for the current snapshot ..... keep polling ├─ GREEN/YELLOW, and snapshot recovery activity │ for the current snapshot was observed ..... SUCCESS └─ RED: │ GET /_cat/shards/{restored-indices}?format=json&h=index,shard,prirep,state (primaries only: prirep=p, state=UNASSIGNED) │ for each UNASSIGNED primary: GET /_cluster/allocation/explain (index, shard, primary=true) ├─ max_retry decider = NO ............... FAILED (ES stopped retrying) ├─ can_allocate = no_valid_shard_copy ... FAILED (for this restore) ├─ disk_threshold decider = NO .......... FAILED (our app cannot free disk — policy) └─ anything else (transient) ............ keep polling plus a bounded limit on how long the "red with no verdict" state may persist (n configurable minutes). ``` One last factual question: is the `max_retry` decider returning `NO` ("shard has exceeded the maximum number of retries") the correct signal that ES has stopped automatic allocation attempts for the current retry cycle, until the retry counter is explicitly reset with `POST /_cluster/reroute?retry_failed=true`? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 26, 2026, 8:16am UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/6 "2026-08-26T08:16:34Z")
Normally I would expect to just `GET _cluster/health` with some of its `wait_for` options, e.g. `wait_for_no_initializing_shards` and `wait_for_status` rather than polling the recovery API. If the `max_retry` decider returns `NO` then the retry counter needs to be reset before it'll try again, but IIRC that happens automatically under some circumstances. Still, yes you can manually reset this counter with `POST /_cluster/reroute?retry_failed` ---
**Author:** ![ciprianamza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ciprianamza/32/148115_2.png) [@ciprianamza](https://discuss.elastic.co/u/ciprianamza)\ **Post date:** [August 26, 2026, 12:49pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/7 "2026-08-26T12:49:16Z")
So, just to make it clear for me, you would suggest to only use the health endpoint? Something like: ```auto GET /_cluster/health/{restored-indices} ?wait_for_status=yellow &wait_for_no_initializing_shards=true &timeout=30s ``` So that the logic becomes: ```auto restore started │ ▼ repeat until deadline │ ├── health reaches YELLOW/GREEN │ └── SUCCESS │ └── timeout (30s) │ └── try again ``` My question here would be: how do I choose a valid overall waiting time, and how do I know the restore cannot progress without intervention, so that I stop waiting early instead of spinning until the deadline? Or would you still combine this loop with the `allocation/explain` check from my earlier post (`max_retry` decider = NO) on the red path, so known-permanent failures stop the wait immediately? I try to understand what approach you would suggest. Thank you for your answers! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 26, 2026, 4:35pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/8 "2026-08-26T16:35:14Z")
To be honest there's no great solution to this level of detailed monitoring. Long-polling the health API in a loop (you can use a longer timeout than 30s) will tell you whether the restore is complete or not but doesn't show much progress information. The recovery API exposes shard-by-shard information about ongoing restore progress but doesn't show all shards. ---
**Author:** ![ciprianamza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ciprianamza/32/148115_2.png) [@ciprianamza](https://discuss.elastic.co/u/ciprianamza)\ **Post date:** [August 27, 2026, 2:08pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/9 "2026-08-27T14:08:23Z")
Thanks for your time! Unfortunately, I'm still not sure how to handle my case. We call the restore request with wait\_for\_completion = false. I'd like to be able to determine whether the restore failed or succeeded. If the health request returns **GREEN** , then I know it succeeded. **YELLOW** means that it succeeded or that it is not yet decided? I assume it means that it succeeded. The problem arises when the response is **RED**. That can mean it failed, but can it also mean it may still succeed later, without manual intervention? In that case I can poll the health endpoint at some interval, but how do I know whether **RED** means it has failed _ **or** _ that it is still trying? We have an 80 GB database, so the process may well take a while. Is the only option to assume some maximum time? The trouble is that if I pick a timeout, I will tell the user the restore failed while the process is in fact still running in the background and might eventually succeed. That is why I'm not sure how to make certain the process has finished 100%, whether the outcome is success or failure. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [August 28, 2026, 11:27am UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/10 "2026-08-28T11:27:00Z")
I smell over engineering here. You are trying for 100%. Accept less. Because even IF you convince yourself you’ve reached 100%, you won’t have. I can’t see how you can validate either, how are you simulating all possible failure scenarios? Your cluster can certainly be red independently of a running/completed restore process. So, at least IMHO, you are mixing concepts. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 30, 2026, 4:03pm UTC](https://discuss.elastic.co/t/how-to-reliably-verify-a-snapshot-restore-succeeded-failed-shards-disappear-from-recovery/389912/11 "2026-08-30T16:03:41Z")
> [@ciprianamza](#): > > but how do I know whether **RED** means it has failed _ **or** _ that it is still trying? I'm not sure you need to distinguish these cases. Even if it eventually finds a way to recover (e.g. you add a new node to the cluster) the fact that it failed even once is already notable and worthy of further investigation.