How does Watcher Alert throttling exactly work?

Elastic Stack Elasticsearch
1

Hello everyone,

I'm wondering how Watcher Alert throttling exactly works. Does the search result have any influence on the throttling? I'm using Watcher with APM and would like to get only one notification per error per day. If I set the throttle period to one day does this mute other erros which might occur within this throttle period?

I'm using the Elasticsearch Service with Elasticsearch 7.5.1.

Thank you for your help.

2

The search result does have an influence, as the throttling mechanism is resetted automatically if the conditions turns false again, once throttling has happened (or the throttling period is expired).

You can read about the thottling logic in the last couple of paragraphs at https://www.elastic.co/guide/en/elasticsearch/reference/7.5/actions.html#actions-ack-throttle

3

Thank you for the explanation @spinscale.

This means that a condition which makes sure that ctx.payload.hits.total is bigger than 0 isn't a very good way for throttling different kind of errors?

What would be the best approach to make sure that only one email per error per day will be sent?

4

Are you sure, that this is what you are after? Why does the existing throttling implementation not work for you?

What if the error occurs twice a day, after it is supposed to be fixed?

How about only running the watch once per day then and have a single watch for each class of errors you want to get alerted only once a day? Even though I personally would not do that.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.