I have created a watcher, integrated it with PagerDuty and it works nicely. But is has two problems:
-
Every time the watcher comes alive to watch, if the error condition is still present, it sends another trigger to PagerDuty.
-
It does not autoresolve
I have read this link: Watchers not auto resolving in Pager Duty
As the answer states, I could just create another watcher to send resolve messages to pager duty. But then a resolve will be sent every time the watcher wakes up and there is no error. What I am looking for is to only send one trigger when the error condition arises (as opposed to whenever it is present, even if a trigger message has already been sent) and one resolve whenever it clears (as opposed to whenever the error condition is absent).
Ideally, I would make the threshold (for backlog in my case) lower for clearing and higher for raising, so that we don't oscillate between error condition and no error condition too often.
If there is a way to save state in a watcher between invocations, then this would be very doable. But I can't find a way to save state.