Watchers not auto resolving in Pager Duty

[Brandondqm (Brandon)]

Hi there

I have built some watchers that fire off to our Pager Duty service, below is an example of a watcher. The alert triggers and works perfectly, however, the issue I am having is that the alert doesn't auto resolve when the threshold is back to normal, I will have to go into pager duty and manually resolve the alert.
I am guessing that I will need to add some sort of 'condition' to my watcher so that it knows when to resolve the alert, but I am not sure how to achieve this, any ideas of what it would look like? Or maybe there a better way of doing this?

  "actions" : {
    "pagerduty": {
      "webhook": {
        "scheme": "https",
        "method": "POST",
        "host": "events.pagerduty.com",
        "port": 443,
        "path": "/generic/2010-04-15/create_event.json",
        "body": "{\"service_key\": \"MyServiceKey\",\"incident_key\": \"MyIncidentKey\",\"event_type\": \"trigger\",\"description\": \"Test\",\"details\":{\"alert_source\":\"watcher\",\"description\":\"Test Alert\",\"LogMessage\":\"Something has gone wrong\"}}",
        "headers": {"Content-type": "application/json"}
      }
    }
  }
}

[spinscale (Alexander Reelsen)]

instead of having a condition for the watch, each action can have a condition. This means, you could have one action that creates a pagerduty alert and another action that marks it as resolved.

You could also have two different watches for this, that might keep the complexity of a single watch down.

--Alex

[Brandondqm (Brandon)]

@spinscale, thank you for your reply. Please can you advise on how this would done? I am still not sure on how the alert will know when the threshold is back to normal so that it fires off an a resolved action.