We have alerting rules in our company which are triggered even though it seems that they shouldnt
in this example i've configured the alert to trigger when then number of docs is below 75k for the last 30mins. and when running the "Test query" i can see that in the last 30m there are over 290k docs. hence the alert should not fire and as you can see it is active.
then i copied the DSL query using "Copy query" and ran in console under <OUR_INDEX>/_search and saw that the number of hits are limited by 10k
then i've modified the threshold of the rule to below 9999 to see if the alert is still triggered.,
and surprisingly it is recovered, which means maybe that it checks the doc count against the limit (10k)?
Please help me figure this out, thanks!