Abnormal Datafeeding detected in Anomaly Detection Jobs

[Souvik_Das]

Hi Team,

There are a couple of machine learning jobs currently running in our ELK environment. With this, we have configured watcher alerts also, based on the record-level anomaly score. Since a few days back, we've been receiving false positive alerts.
After investigation, we found that, though there are data in the index checked from discover (540 hits) for that timeframe, the anomaly detection result window is showing actual 0. Please see below for reference --

Now, by further deep-diving, we found that datafeed is not able to pull data properly, see the screenshot below for reference --

Why datafeed is not able to pull these many documents?

FYI, current settings are as below --

query_delay - 240s
scroll_size - 1000

Also, let me know if you need any other information.

Regards,
Souvik

[richcollier]

if this is an ongoing issue you have to either increase your query_delay so that you are not missing data or figure out why it takes so long (apparently longer than 2 minutes) for your data to be ingested and searchable (by the ML job)