# Datafeed has missed documents due to ingest latency error **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [August 23, 2020, 12:32pm UTC](https://discuss.elastic.co/t/datafeed-has-missed-documents-due-to-ingest-latency-error/246016 "2020-08-23T12:32:05Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\ **Post date:** [August 23, 2020, 12:32pm UTC](https://discuss.elastic.co/t/datafeed-has-missed-documents-due-to-ingest-latency-error/246016/1 "2020-08-23T12:32:06Z")
Hello, I have many of these in the Job Management \> Job Messages tab, and I'm looking for a way to check it. first of all, the dates don't make since, but I guess it's a timezone bug in Kibana (first column's date is lower than the date in the message itself) second, is there any way to validate ingestion time? I don't think that we have a 30 minutes delay - current query\_delay setting in the datafeed (It still may be true though...) ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/1/a1528fea073de3f6318752250c67659b56152049.png) ---
**Author:** ![sophie\_chang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sophie_chang/32/18008_2.png) [@sophie\_chang](https://discuss.elastic.co/u/sophie_chang)\ **Post date:** [August 24, 2020, 8:37am UTC](https://discuss.elastic.co/t/datafeed-has-missed-documents-due-to-ingest-latency-error/246016/2 "2020-08-24T08:37:45Z")
Hi @liorg2 The left hand timestamp in the job messages tab is displayed in your browser timezone by default. You can change this in Kibana Management / Advanced Settings (although I would not think you'd want to as it affects all your visualizations too). The backend anomaly detection uses UTC as this is a log message from the backend. This log message is a string, rather than a field with a data type of timestamp, so it is unformatted. The time reported in the message is the start of the bucket, and the end of the bucket is this start time plus the `bucket_span` configured for the job. This recent blog describes how you can add an ingest time to your ingest pipeline. [https://www.elastic.co/blog/calculating-ingest-lag-and-storing-ingest-time-in-elasticsearch-to-improve-observability](https://www.elastic.co/blog/calculating-ingest-lag-and-storing-ingest-time-in-elasticsearch-to-improve-observability) This uses the `set` ingest processor [https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html) More information on delayed data can be found here [https://www.elastic.co/guide/en/machine-learning/7.9/ml-delayed-data-detection.html](https://www.elastic.co/guide/en/machine-learning/7.9/ml-delayed-data-detection.html) Hope this helps Sophie ---
**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\ **Post date:** [August 24, 2020, 1:27pm UTC](https://discuss.elastic.co/t/datafeed-has-missed-documents-due-to-ingest-latency-error/246016/3 "2020-08-24T13:27:20Z")
thanks for the information! just few yes/no questions... currently I don't have any injection processor, will it make a load on the cluster? do I need to add nodes for that? (I use elastic cloud..) thank Lior ---
**Author:** ![sophie\_chang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sophie_chang/32/18008_2.png) [@sophie\_chang](https://discuss.elastic.co/u/sophie_chang)\ **Post date:** [August 24, 2020, 2:24pm UTC](https://discuss.elastic.co/t/datafeed-has-missed-documents-due-to-ingest-latency-error/246016/4 "2020-08-24T14:24:09Z")
Hi You need a node with an ingest role `i`. You should have one already in cloud. ```auto # GET _cat/nodes 127.0.0.1 70 80 1 0.00 0.03 0.05 dlm - node1 127.0.0.1 46 80 1 0.00 0.03 0.05 dlmt * node2 127.0.0.1 63 80 1 0.00 0.03 0.05 dilmt - node3 ``` With respect to load, please benchmark locally to see if there is a possible performance impact. You don't really need the painless script part from the blog example, and I doubt that the single `set` processor will make a noticeable difference - but it depends on your data and use case. Before you change your ingest, as a first step you could also try a manual validation that you have delayed data by running a search that will replicate what the delayed data check is trying to achieve. Assuming you have a `15m` `bucket_span` and a `30m` `query_delay`, create a date histogram search e.g. count of events every 15m from `now-90m` say. Manually refresh this periodically over the course of the next 90m and see if the counts change as time elapses. Pay particular attention to the counts from time buckets that are greater than 30m ago. If these are changing, this suggests an ingest latency. ---
**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\ **Post date:** [August 24, 2020, 2:42pm UTC](https://discuss.elastic.co/t/datafeed-has-missed-documents-due-to-ingest-latency-error/246016/5 "2020-08-24T14:42:33Z")
Make sense I'll check that Thanks a lot ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 21, 2020, 2:42pm UTC](https://discuss.elastic.co/t/datafeed-has-missed-documents-due-to-ingest-latency-error/246016/6 "2020-09-21T14:42:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.