How to put condition on a failed watch?

Elastic Stack Elasticsearch
1

I have a watch set up on cluster health, where I trigger an action if the status is red. However, it's possible that the node I am watching go down. In this case, the watch history will record a failed attempt and take no action. Is there a way to trigger an action if the watch attempt fails?

Also, somewhat off topic from the original question, but is it possible to set the condition so that it's 'red' OR 'yellow'?

2

Hey,

first on your first question. The simplest way would be to check not for green using the not_eq operator, as red/yellow/green are all the possible states.

On your second question: Right now the watch itself does have no mechanism to take an action if the input is already failure. You could however query the watch history for failed watch executions and thus find out which watches may have already failed that early.

--Alex

3

Thanks for the info. It's unfortunate that there's no action trigger for failures. It seems like a reasonably common failover situation. So based on your suggestion, I would be querying against the watch history for the particular watch that may fail, and check within the timespan I wish to look for (e.g. triggered_time - 15m to triggered_time). Is that right?

Also, on using the not_eq... if I query against a field with 4+ enumerations, using not_eq is not a good alternative to an OR statement. It works for cluster state because there's only 3 possible values. Is your answer implying that multiple conditions are not possible in watcher?

Thanks again :slight_smile:

4

Hey,

first: Yes, that sounds right... Maybe add one monitoring watch the checks for failed executions, instead of a one monitor watch per regular watch.

second: if you need more fancy logic of comparing things, you should go with a script condition (see here, note this for 5.0, you may want to check the 2.4 watcher docs), which allows you to do arbitrary comparisons like the one you need or go more fancy and compare two different search results with each other.

--Alex

5

It looks like 2.4 does support the same scripts. I'll take a look. Was hoping the compare operator had some creature comforts like contains and contains_all! :slight_smile: anyways, thanks for the guidance! have a good one!