Snooze or Dismiss Watcher Alert

Elastic Stack Elasticsearch
1

Is there a way to disable watcher alerts

A use case will be like
Initially alert set based on one month worth of logs

  1. Alert came in
  2. Read the alert
  3. Take note of the alert and read the relevant logs
  4. Ignore this alert but still want to be alerted on new occurances

How can the above be achieved?

2

Hey,

you might want to check the Ack Watch API and the Deactivate Watch API, depending on what you want to do.

If you only want to execute an alert once, you could try to deactivate it as part of the action using a webhook.

Hope this helps, if not maybe just clarify by explaining your use-case a bit more.

--Alex

3

Hello,

@spinscale, I am having a similar issue.
I am trying to use the deactivate watch API in a webhook action to deactivate (triggering) watch.
For example, the action in the following fails for me:

PUT /_watcher/watch/epm21_down
{
  "trigger": {
    "schedule": { "interval": "5m" }
  },
      //cut the queries + condition for brevity 
  "actions": {
    "deactive_current": {
      "webhook": {
        "method": "PUT",
        "host": "localhost",
        "port": 9200,
        "path": "/_watcher/watch/{{ctx.watch_id}}/_deactivate"
        }
     }
   }
}

When I try to execute this the return reads:

...
      "actions": [
        {
          "id": "deactive_current",
          "type": "webhook",
          "status": "failure",
          "reason": "received [500] status code",
          "webhook": {
            "request": {
              "host": "localhost",
              "port": 9200,
              "scheme": "http",
              "method": "put",
              "path": "/_watcher/watch/epm21_down/_deactivate"
            },
            "response": {
              "status": 500,
              "headers": {
                "Content-Length": [
                  "603"
                ],
                "Content-Type": [
                  "application/json; charset=UTF-8"
                ]
              }
            }
          }
        }
      ]

The (relevant) output in the elasticsearch log is:

[2016-03-15 11:36:24,734][WARN ][watcher.watch            ] [vantgvmlnx089] failed to acquire lock on watch [epm21_down] (waited for [10 seconds]). It is possible that for some reason this watch execution is stuck
[2016-03-15 11:36:24,736][INFO ][rest.suppressed          ] /_watcher/watch/epm21_down/_deactivate Params: {id=epm21_down}
ElasticsearchTimeoutException[could not ack watch [{}] within [{}]... wait and try again. If this error continues to occur there is a high chance that the watch execution is stuck (either due to unresponsive external system such as an email service, or due to a bad script]
        at org.elasticsearch.watcher.WatcherService.setWatchState(WatcherService.java:237)
        at org.elasticsearch.watcher.WatcherService.deactivateWatch(WatcherService.java:230)
        at org.elasticsearch.watcher.transport.actions.activate.TransportActivateWatchAction.masterOperation(TransportActivateWatchAction.java:66)
        at org.elasticsearch.watcher.transport.actions.activate.TransportActivateWatchAction.masterOperation(TransportActivateWatchAction.java:41)
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:157)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2016-03-15 11:36:24,743][WARN ][watcher.actions.webhook  ] [vantgvmlnx089] received http status [500] when connecting to watch action [epm21_down/webhook/deactive_current]

Do you know what's going on?

Thanks,
Michel