I've read this nice post on how you could possibly set up Watcher to trigger an action only when a watch change state (similar to how cloudwatch does it with events). I'd love to set something like that up to avoid spamming my alert channels and also get some feedback when things go back to normal (I once had a situation when the whole cluster was not doing very well and alert were not sent as a result).
As I understand the general idea is:
- have my watches check whatever condition I check already
- add chained input with watch history
- script the condition so that it doesn't only take the metric I look at into account but also watch history to prevent firing if the state is already "error"
Alternatively, would this be possible:
- I keep my current watches exactly as they are, but remove notification action
- add a new watch per each of the above that looks at state changes in watcher history and triggers action on state transition
Does that make sense? Is there a more up-to date version of how to set something like that up? I use 7.8 and it seems like index names changed a bit and debugging watcher is a nightmare, so some verified instructions would be appreciated.