# Machine learning jobs not reflecting new data **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [March 30, 2018, 8:58pm UTC](https://discuss.elastic.co/t/machine-learning-jobs-not-reflecting-new-data/126283 "2018-03-30T20:58:03Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![David\_Tomasheski](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@David\_Tomasheski](https://discuss.elastic.co/u/David_Tomasheski)\ **Post date:** [March 30, 2018, 8:58pm UTC](https://discuss.elastic.co/t/machine-learning-jobs-not-reflecting-new-data/126283/1 "2018-03-30T20:58:03Z")
I have a few machine learning jobs running all on the same index pattern. They are single metric doc count jobs. I am getting new data in but my count in the Single Metric Viewer flat lines to 0. I have no idea what I'm doing wrong. They are not marked as not receiving new data just doesn't show it. Any advice would be greatly appreciated. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [March 30, 2018, 9:14pm UTC](https://discuss.elastic.co/t/machine-learning-jobs-not-reflecting-new-data/126283/2 "2018-03-30T21:14:59Z")
Hi David, I assume that if you were to run the ML job over historical data, you don't get this problem (it's only with live data)? If that's the case, the most likely culprit is that your ingest of your data is delayed to the point that when ML "looks for it" (i.e. ML looking for the data in the last X minutes, which is equal to the bucket\_span) - the data is not yet there - but is there at a later time. You can check to see how many events the ML job is seeing by looking into the .ml-anomalies-\* index for your `job_id` and for `result_type:bucket` - and inspect the `event_count` field. Here, you can see a consistent 870 events per bucket. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4c3ba15978bb3100c71a93bb0e7c7f077a7f4693.png) So, if you're finding that only your "live data" is getting missed, you probably need to increase the `query_delay` parameter of the datafeed configuration: ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/d/eda710532027d503ebca268bfde8aebc871b4d81.png) ...or figure out why things are not getting ingested/indexed as fast as they could be. ---
**Author:** ![David\_Tomasheski](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@David\_Tomasheski](https://discuss.elastic.co/u/David_Tomasheski)\ **Post date:** [April 3, 2018, 5:04pm UTC](https://discuss.elastic.co/t/machine-learning-jobs-not-reflecting-new-data/126283/3 "2018-04-03T17:04:29Z")
This seems to be my problem. My data doesn't come in continuously. So I'll get a burst of data starting at about 10 after the hour and that will last until about 10ish mins. This data reflects the previous hour so I'll get data for 9:00-10:10 in one big burst from about 10:10-10:20. Is there a good way to configure ML for this kind of setup. As this is external data there is no way for me to change the delivery to a more constant method. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [April 3, 2018, 6:26pm UTC](https://discuss.elastic.co/t/machine-learning-jobs-not-reflecting-new-data/126283/4 "2018-04-03T18:26:45Z")
So one option would be to do a 1 hour `bucket_span` and a `query_delay` of 30m. That would mean that at 10:30, ML would ask for, and analyze data from 9:00-10:00. Then at 11:30, ML would would ask for, and analyze data from 10:00-11:00, and so on. If you desire a smaller `bucket_span` for analysis reasons (let's say 10m), then you'd have to increase `query_delay` to something like 90m. So that in this case, at 10:30, ML would ask for, and analyze data from 9:00-9:10 and then at 10:40, ML would ask for, and analyze data from 9:10-9:20, and so on. In both of the above cases, make the `frequency` parameter equal to the `bucket_span`. For data that is more real-time, often the `frequency` parameter is a fractional value of the `bucket_span`. (See [https://www.elastic.co/guide/en/elasticsearch/reference/6.2/ml-put-datafeed.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/ml-put-datafeed.html) for more info). But, in your case, it doesn't really make sense for `frequency` to be anything other than equal to the `bucket_span`. Hope that helps ---
**Author:** ![David\_Tomasheski](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@David\_Tomasheski](https://discuss.elastic.co/u/David_Tomasheski)\ **Post date:** [April 4, 2018, 3:16pm UTC](https://discuss.elastic.co/t/machine-learning-jobs-not-reflecting-new-data/126283/5 "2018-04-04T15:16:46Z")
Thanks this seems to have done the trick! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 30, 2018, 10:21pm UTC](https://discuss.elastic.co/t/machine-learning-jobs-not-reflecting-new-data/126283/6 "2018-10-30T22:21:32Z")