This sounds eerily similar to an older bug (ML alerts triggering on interim result) that existed about 2 years ago (which was fixed).
That bug centered around an incorrect calculation of an anomaly, expressed as an interim result (is_interim:true
). An interim result is a result that exists, but the current time bucket is not yet complete - and the anomaly score could change by the time the current time bucket is closed (resulting in the record how having is_interim:false
). The fact that the document that was an anomaly is now gone would be consistent with a situation where the interim result was deemed anomalous, but then the bucket was closed, it was no longer considered anomalous (and thus the record disappears).
I'm not saying that there couldn't be another bug in this area - there very well may be - because we shouldn't calculate an anomalous interim result for a count
of 0
Until we figure out there's a bug here, perhaps you can avoid this situation by adding the following qualifier to your Watcher query:
{ "term" : { "is_interim" : "false"}}
in this way, you avoid looking at interim results.