Watcher alert only on state transition (error / ok state)

Elastic Stack Elasticsearch
1

I've read this nice post on how you could possibly set up Watcher to trigger an action only when a watch change state (similar to how cloudwatch does it with events). I'd love to set something like that up to avoid spamming my alert channels and also get some feedback when things go back to normal (I once had a situation when the whole cluster was not doing very well and alert were not sent as a result).

As I understand the general idea is:

  • have my watches check whatever condition I check already
  • add chained input with watch history
  • script the condition so that it doesn't only take the metric I look at into account but also watch history to prevent firing if the state is already "error"

Alternatively, would this be possible:

  • I keep my current watches exactly as they are, but remove notification action
  • add a new watch per each of the above that looks at state changes in watcher history and triggers action on state transition

Does that make sense? Is there a more up-to date version of how to set something like that up? I use 7.8 and it seems like index names changed a bit and debugging watcher is a nightmare, so some verified instructions would be appreciated.

2

Hey,

that sounds about right. One of the advantages of the first idea would be that a single watch cannot collide with another watch, that probably tries to clean things up, so I would probably go with that, despite being somewhat more complex.

With regards to better and easier debugging of watches: Take a look a this blogpost that I wrote a while ago - while the endpoints do not have the _xpack suffix anymore, most of this is still accurate.

if you have questions, just drop them in the forum :slight_smile:

--Alex

3

heh, this way to cumbersome to set up than it should be. Debugging "painless" is also everything but "painless". I'm wondering if watcher will continue to evolve or the latest improvements are the most we can expect...

4

See the following links regarding that topic and the new alerting framework within the Elastic Stack



5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.