Snooze or Dismiss Watcher Alert

Elastic Stack Elasticsearch
3

Hello,

@spinscale, I am having a similar issue.
I am trying to use the deactivate watch API in a webhook action to deactivate (triggering) watch.
For example, the action in the following fails for me:

PUT /_watcher/watch/epm21_down
{
  "trigger": {
    "schedule": { "interval": "5m" }
  },
      //cut the queries + condition for brevity 
  "actions": {
    "deactive_current": {
      "webhook": {
        "method": "PUT",
        "host": "localhost",
        "port": 9200,
        "path": "/_watcher/watch/{{ctx.watch_id}}/_deactivate"
        }
     }
   }
}

When I try to execute this the return reads:

...
      "actions": [
        {
          "id": "deactive_current",
          "type": "webhook",
          "status": "failure",
          "reason": "received [500] status code",
          "webhook": {
            "request": {
              "host": "localhost",
              "port": 9200,
              "scheme": "http",
              "method": "put",
              "path": "/_watcher/watch/epm21_down/_deactivate"
            },
            "response": {
              "status": 500,
              "headers": {
                "Content-Length": [
                  "603"
                ],
                "Content-Type": [
                  "application/json; charset=UTF-8"
                ]
              }
            }
          }
        }
      ]

The (relevant) output in the elasticsearch log is:

[2016-03-15 11:36:24,734][WARN ][watcher.watch            ] [vantgvmlnx089] failed to acquire lock on watch [epm21_down] (waited for [10 seconds]). It is possible that for some reason this watch execution is stuck
[2016-03-15 11:36:24,736][INFO ][rest.suppressed          ] /_watcher/watch/epm21_down/_deactivate Params: {id=epm21_down}
ElasticsearchTimeoutException[could not ack watch [{}] within [{}]... wait and try again. If this error continues to occur there is a high chance that the watch execution is stuck (either due to unresponsive external system such as an email service, or due to a bad script]
        at org.elasticsearch.watcher.WatcherService.setWatchState(WatcherService.java:237)
        at org.elasticsearch.watcher.WatcherService.deactivateWatch(WatcherService.java:230)
        at org.elasticsearch.watcher.transport.actions.activate.TransportActivateWatchAction.masterOperation(TransportActivateWatchAction.java:66)
        at org.elasticsearch.watcher.transport.actions.activate.TransportActivateWatchAction.masterOperation(TransportActivateWatchAction.java:41)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:157)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2016-03-15 11:36:24,743][WARN ][watcher.actions.webhook  ] [vantgvmlnx089] received http status [500] when connecting to watch action [epm21_down/webhook/deactive_current]

Do you know what's going on?

Thanks,
Michel