Datafeed has been retrieving no data for a while

Elastic Stack Elasticsearch
1

Hello everybody,

i just created some anomaly detection jobs, however new records are not processed after the lookback was performed.
New data is available in the source index the jobs are working on.
If i reset the job and start the datafeed of it again, the new records are taken into account while the other jobs do not process anything beyond their lookback period.

Hence they are flagged with "Datafeed has been retrieving no data for a while".

Thanks in advance!

2

A likely cause of this is that the query_delay on the datafeed is too low.

A datafeed will only search a particular time period once, and does it slightly behind real-time to allow time for the data to be indexed. How far behind real-time is determined by query_delay.

For example, what you are describing could be explained if your data is indexed 5 minutes after its @timestamp but your query_delay is only 2 minutes.

The problem might be something more complex, but that's the most obvious thing to check first.

1 Like
3

Thank you for your quick response!
I increased the query_delay for one of the jobs and will see, if it fixes it.

In the meantime:
I see, in the "View datafeed counts" graph of the job the correct count of documents in the source index. Thus, the job is somehow "aware" of the presence of the newer documents, i assume.
Quite confusing.

Could this be also caused by repeated deletion and creation of anomaly jobs? I was experimenting quite a bit with it.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.