Datafeed missing in ML job

[lcawley (Lisa Cawley)]

Hello! I think there is some good applicable advice in Datafeed has missed documents due to ingest latency error

In particular, there are references there to

Handling delayed data | Machine Learning in the Elastic Stack [7.9] | Elastic

and Calculating ingest lag and storing ingest time in Elasticsearch to improve observability | Elastic Blog

Sophie also provided the following advice, which I think is applicable here too:

Before you change your ingest, as a first step you could also try a manual validation that you have delayed data by running a search that will replicate what the delayed data check is trying to achieve. Assuming you have a 15m bucket_span and a 30m query_delay , create a date histogram search e.g. count of events every 15m from now-90m say. Manually refresh this periodically over the course of the next 90m and see if the counts change as time elapses. Pay particular attention to the counts from time buckets that are greater than 30m ago. If these are changing, this suggests an ingest latency.