+1 for automatic acknowledgement in watcher

[casieowen]

Hi,

I see a couple questions from last year about the capability to automatically acknowledge a watch within the definition of a watch itself (including links to those questions below, for reference). Are there any updates on that as a feature?

And/or, would it be possible to do it as a webhook action?

Generic watch, specific alerts, duplicate alerting?

Imagine that I have 100 machines to monitor and that each machine is named by its number (i.e. the first machine is named ("1"). Imagine that I wish to alert if the disk usage is over 90% on any of the machines. Imagine that LogStash is inserting data into ES every minute with the current disk percentage for each machine. I would like to send an alert to pagerduty for every machine that is over 90% disk usage. I know how to craft my search query to find machines with disk usage over 90%. Su…

How can I get Automatic Acknowledgement-based Throttling?

This page: https://www.elastic.co/guide/en/watcher/current/watch-acknowledgment-throttling.html suggests that an action will only be executed once (until the condition becomes false and then true again). I would like to get that behavior. However, I'm not clear on how I would achieve this "one alert per condition". I viewed this page: https://www.elastic.co/guide/en/watcher/current/actions.html. When I read that page, I see that can manually "ack" an alert, but I want the alert to work aut…

Thanks,
Casie

[skearns (Steve Kearns)]

Hi Cassie,

Can you share a bit about your use-case and how you would use an auto-acknowledge feature vs. the existing throttling capabilities?

Thanks,
Steve

[casieowen]

We ingest incident data into a single incidents index. If a document with a priority 1 incident is created in the index, the document will keep it's priority 1 status and its "Open" status for an indefinite amount of time. We want to query this index for any P1/open incidents but only ever alert one time. Currently I have it set to query the index for any P1/open incidents created in the last twenty minutes, and I have a 30 minute throttle period defined, and that will work ok except for if there is a delay in ingesting data. For example: a P1 incident is created at 10 AM today, but something in our ingestion pipeline fails, we don't ingest any data between 10-11AM, we get the data in there at 11:00. . .the watch looks for anything from within the last 20 minutes, and doesn't find any.

Does that help?

[casieowen]

@casieowen

but something in our ingestion pipeline fails, we don't ingest any data between 10-11AM, we get the data in there at 11:00. . .the

A couple other things I'm thinking might work:

1. Can a watch modify data in the index itself? So, if we added a field for "alerted", could the watch populate that field?

2. We're currently indexing the payload of the watch query into a watch_p1 index. I'm going to see if I can define the watch query to look for an open, P1 incident for which there is no document in the watch_p1 index (since that would only get created by a watch execution that would have already alerted).

Let me know if you have any other thoughts and/or what plans are re: auto acknowledgement for watches.

Thanks!
Casie

[warkolm (Mark Walkom)]

This is definitely something you should also ask your Elastic support engineer

[casieowen]

Yeah, I was thinking of doing that, but since I saw other people asking for this feature here in the community. . .

But, yes, Mike, here I come. Mike is awesome all the time, so that helps.

Casie