Machine Learning datafeed skipping documents that seem to be there

[richcollier]

I think there might be a few things going on here.

1. There is a bug that was introduced in v6.5 (and will be fixed in v6.6.2+) that inadvertently creates an anomaly on an interim (un-finalized, or still-open bucket). See:

ML alerts triggering on interim result

After upgrading from 6.3 to 6.5.0 I've been getting Watcher alerts from ML jobs that are false positives. If I click on the alert link fairly quickly I see 0 hits when there should be some large number, but it also says "Interim result". If I wait a few minutes and refresh, the anomaly score goes from 99 down to <1 as it's found more than 0 hits. Is this an indication of a performance issue on my ES cluster or some changed behavior in ES - ML - Watcher interactions with 6.5?

and the corresponding bug:

[ML] Unexpected interim results after advancing time into new empty bucket

2. The auto-annotation for missing data, however, should not stumble onto this bug because it explicitly ignores interim buckets. In order to validate your datafeed timing (what bucket's it's querying and when), you could enable TRACE logging for the datafeed:

PUT _cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.ml.datafeed": "TRACE"
  }
}

(this is a transient setting that won't survive a cluster re-start but you can always reset this back to "DEBUG" or "NORMAL" when this experiment is over)

You can also have your Watch log what it sees as well - then, in the elasticsearch.log file we should have a better understanding of when the datafeed runs and what window of time it queries - while at the same time seeing the output of your watch that is trying to also do the validation.