I'm trying to get my Watcher to not return an alert right away but wait a moment to confirm or deny the problem.
So instead of sending an alert directly, I want the Watcher to wait for example 15 minutes then look again and send the alert if the problem persists.
I think you can write the first alert to another index with the timestamp.
Then add a watch on this index and if the number of hits within an 15 minutes period is more than one, send the alert you want.
How about refining the way you are querying Elasticsearch? Instead of one query looking back the last 15 minutes, how about creating two queries. The first looks back from now till 15 minutes ago, the second from 15 minutes ago till 30 minutes ago (or whatever time delay you are interested in) and then you check if the problem exists in both queries. If that is the case trigger an alert.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.