I'm experiencing an issue with the behavior of Kibana Rules.
I have Metricbeat writing different metrics to Elasticsearch every 10 seconds.
I want to create rules using Elasticsearch Query over the values that are being inserted.
i.e. I want to be alerted when a specific value drops below 90.
I configured the rule to run every 5 seconds for fast notifications.
What i'm experiencing is:
- The rule is triggered and a notification is sent.
- After 5 seconds, the next time the rule is querying the condition, it's being recovered, even though no new data inserted to Elasticsearch.
So after another 5 seconds (when Metricbeat is indexing new data), if the value is still below 90, i will be notified again, even though it wasn't actually recovered.
And when the value will actually be recovered, i wouldn't be aware of it, since it was "auto-recovered" after 5 seconds.
Any idea how to get around this issue?
Thanks in advance!