# Best practices for monitoring curator snapshot (failures)
**URL:** https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386
**Category:** Elasticsearch
**Created:** [November 27, 2018, 4:55pm UTC](https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386 "2018-11-27T16:55:03Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@jonassteinberg1](https://discuss.elastic.co/u/jonassteinberg1)
#### Post date: [November 27, 2018, 4:55pm UTC](https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386/1 "2018-11-27T16:55:03Z")
@theuntergeek
I have a nice curator --\> AWS S3 snapshot pipeline for backing up my clusters, but from time-to-time a snapshot will fail and thus I'm wondering what the best way to monitor snapshot failures is?
Right now I'm rsyslogging all of my curator events to my cluster and I was planning on using watchers to email based on certain conditions (like failures). Rsyslog works great and I've no doubt the watchers will technically work...but I don't know what the best thing to scrape for would be? I've thought about watching for the literal exception strings thrown on the various snapshot failure types and I think that will work to a certain extent. But my concern is if I update curator or ES or whatever and those exception messages change then my watchers will break. If you have any good ideas of what to 'watch' for or a totally different approach not involving watchers or log parsing I'd be very interested to know your thoughts.
Thanks,
Jonas Steinberg
---
### Author:  [@jonassteinberg1](https://discuss.elastic.co/u/jonassteinberg1)
#### Post date: [November 27, 2018, 5:19pm UTC](https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386/2 "2018-11-27T17:19:29Z")
Small tangent, but related:
My curator jobs run on cron. I _had_ thought of just sending an email every time curator's exit was !0 but even when I break the jobs intentionally curator still exits 0. Thoughts?
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [November 27, 2018, 5:27pm UTC](https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386/3 "2018-11-27T17:27:29Z")
Curator does perform a [check for `SUCCESS` state](https://github.com/elastic/curator/blob/v5.6.0/curator/actions.py#L1488-L1499), which runs [immediately after](https://github.com/elastic/curator/blob/v5.6.0/curator/actions.py#L1536) it has detected that the snapshot has completed. This check is only performed, however, if `wait_for_completion` is set to `true`.
If an exception is not causing a non-zero exit code, do you have `continue_if_exception` set to `true`? Otherwise, this sounds like a bug.
---
### Author:  [@jonassteinberg1](https://discuss.elastic.co/u/jonassteinberg1)
#### Post date: [November 27, 2018, 5:59pm UTC](https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386/4 "2018-11-27T17:59:01Z")
@theuntergeek
1. Yes I have `wait_for_completion` set to `true`, so regarding an expression or literal to 'watcher' for: what would you recommend? It would be nice if the watcher _only looked for the SUCCESS state only after the snapshot either completes or breaks_? I'm not asking you to write the watcher, but merely an expression to use or something like that.
2. `continue_if_exception` is set to its default of `false`. I've looked into this situation more and it seems that curator is working reasonably.
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [November 27, 2018, 6:27pm UTC](https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386/5 "2018-11-27T18:27:04Z")
While a snapshot is running, its state is `IN_PROGRESS`. Curator will [continue to check this](https://github.com/elastic/curator/blob/v5.6.0/curator/utils.py#L1523-L1560) until a different state is reported, any of `SUCCESS`, `PARTIAL`, `FAILED`, or anything else not `IN_PROGRESS`. The final state is checked again, and non `SUCCESS` states result in an exception being raised.
If you want it to be reported on, you could make API calls with watches to check for a snapshot in progress, and to see if any of the previous snapshots finished with a state other than `SUCCESS`.
A future release of Curator will log each action's results to an index, which will be easier to use with watches. But that's not ready for consumption yet.
---
### Author:  [@jonassteinberg1](https://discuss.elastic.co/u/jonassteinberg1)
#### Post date: [November 27, 2018, 6:40pm UTC](https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386/6 "2018-11-27T18:40:17Z")
Will do!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 25, 2018, 6:40pm UTC](https://discuss.elastic.co/t/best-practices-for-monitoring-curator-snapshot-failures/158386/7 "2018-12-25T18:40:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.