I am wondering about a particular use case. I am using watcher to send a webhook to an external alert tracking mechanism when a watch condition is met. The external system receives the event and creates an incident ticket in our enterprise service desk. This is pretty easy to set up with watcher
Our policies dictate that when an alert ( in this case my watch ) is restored, the alerting system send a "stand down" event to the helpdesk which will close down the incident ticket. Since watches dont maintain an internal state over time, I'm wondering how to implement this.
Right now I am considering having two watches set up per condition, one for detecting "system down" events then another detecting "system ok" events. Each watch would send either the "system ok" or "system down" event to the same endpoint and update the ticket status accordingly.
To me this seems a bit kludgy and im wondering if anyone here has any thoughts on how to achieve this or has done something similar in their own implementation.