# Any ML job I create runs for the first time and then stops and never runs **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [June 14, 2018, 4:04pm UTC](https://discuss.elastic.co/t/any-ml-job-i-create-runs-for-the-first-time-and-then-stops-and-never-runs/135962 "2018-06-14T16:04:48Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![kaushiksoft](https://avatars.discourse-cdn.com/v4/letter/k/49beb7/32.png) [@kaushiksoft](https://discuss.elastic.co/u/kaushiksoft)\ **Post date:** [June 14, 2018, 4:04pm UTC](https://discuss.elastic.co/t/any-ml-job-i-create-runs-for-the-first-time-and-then-stops-and-never-runs/135962/1 "2018-06-14T16:04:48Z")
Symptom - Any ML job I create runs for the first time and then stops and never runs. Here is an example Job - { "job\_id": "tibco-celect", "job\_type": "anomaly\_detector", "job\_version": "6.0.0", "description": "Celect Trends", "create\_time": 1528730826156, "finished\_time": 1528730828213, "analysis\_config": { "bucket\_span": "15m", "summary\_count\_field\_name": "doc\_count", "detectors": [{ "detector\_description": "non\_zero\_count", "function": "non\_zero\_count", "detector\_rules": [], "detector\_index": 0 } ], "influencers": [] }, "data\_description": { "time\_field": "timestamp", "time\_format": "epoch\_ms" }, "model\_plot\_config": { "enabled": true }, "model\_snapshot\_retention\_days": 1, "model\_snapshot\_id": "1528823700", "results\_index\_name": "custom-tibco-celect", "data\_counts": { "job\_id": "tibco-celect", "processed\_record\_count": 128, "processed\_field\_count": 128, "input\_bytes": 5577, "input\_field\_count": 128, "invalid\_date\_count": 0, "missing\_field\_count": 0, "out\_of\_order\_timestamp\_count": 0, "empty\_bucket\_count": 2, "sparse\_bucket\_count": 0, "bucket\_count": 128, "earliest\_record\_timestamp": 1528689731828, "latest\_record\_timestamp": 1528810168631, "last\_data\_time": 1528810335774, "latest\_empty\_bucket\_timestamp": 1528705800000, "input\_record\_count": 128 }, "model\_size\_stats": { "job\_id": "tibco-celect", "result\_type": "model\_size\_stats", "model\_bytes": 66506, "total\_by\_field\_count": 3, "total\_over\_field\_count": 0, "total\_partition\_field\_count": 2, "bucket\_allocation\_failures\_count": 0, "memory\_status": "ok", "log\_time": 1528823850000, "timestamp": 1528823700000 }, "datafeed\_config": { "datafeed\_id": "datafeed-tibco-celect", "job\_id": "tibco-celect", "query\_delay": "60ms", "frequency": "150s", "indices": ["tibauditxml-_"], "types": ["doc"], "query": { "bool": { "must": [{ "query\_string": { "query": "_", "fields": [], "type": "best\_fields", "default\_operator": "or", "max\_determinized\_states": 10000, "enable\_position\_increments": true, "fuzziness": "AUTO", "fuzzy\_prefix\_length": 0, "fuzzy\_max\_expansions": 50, "phrase\_slop": 0, "analyze\_wildcard": true, "escape": false, "boost": 1 } }, { "match\_phrase": { "EventDescription": { "query": "From Celect: Message sent to sterling successfully", "slop": 0, "boost": 1 } } } ], "adjust\_pure\_negative": true, "boost": 1 } }, "aggregations": { "buckets": { "date\_histogram": { "field": "timestamp", "interval": 900000, "offset": 0, "order": { "\_key": "asc" }, "keyed": false, "min\_doc\_count": 0 }, "aggregations": { "timestamp": { "max": { "field": "timestamp" } } } } }, "scroll\_size": 1000, "chunking\_config": { "mode": "manual", "time\_span": "900000000ms" }, "state": "stopped" }, "state": "opened", "node": { "id": "w-t1K8AzSmm5UBwmQ7FpWg", "name": "w-t1K8A", "ephemeral\_id": "4VahUs23RTGO0js5rRT\_Gw", "transport\_address": "10.75.19.172:9300", "attributes": { "ml.max\_open\_jobs": "10", "ml.enabled": "true" } }, "open\_time": "260859s" } Job logs - 2018-06-11 11:27:06 w-t1K8A Job created 2018-06-11 11:27:06 w-t1K8A Loading model snapshot [N/A], job latest\_record\_timestamp [N/A] 2018-06-11 11:27:06 w-t1K8A Opening job on node [{w-t1K8A}{w-t1K8AzSmm5UBwmQ7FpWg}{4VahUs23RTGO0js5rRT\_Gw}{10.75.19.172}{10.75.19.172:9300}{ml.max\_open\_jobs=10, ml.enabled=true}] 2018-06-11 11:27:07 w-t1K8A Starting datafeed [datafeed-tibco-celect] on node [{w-t1K8A}{w-t1K8AzSmm5UBwmQ7FpWg}{4VahUs23RTGO0js5rRT\_Gw}{10.75.19.172}{10.75.19.172:9300}{ml.max\_open\_jobs=10, ml.enabled=true}] 2018-06-11 11:27:07 w-t1K8A Datafeed started (from: 2018-06-11T03:26:40.100Z to: 2018-06-11T15:26:40.102Z) 2018-06-11 11:27:08 w-t1K8A Datafeed lookback completed 2018-06-11 11:27:08 w-t1K8A Datafeed stopped 2018-06-11 11:27:08 w-t1K8A Job is closing 2018-06-11 11:27:22 w-t1K8A Loading model snapshot [1528730828] with latest\_record\_timestamp [2018-06-11T15:14:49.000Z], job latest\_record\_timestamp [2018-06-11T15:14:49.575Z] 2018-06-11 11:27:22 w-t1K8A Opening job on node [{w-t1K8A}{w-t1K8AzSmm5UBwmQ7FpWg}{4VahUs23RTGO0js5rRT\_Gw}{10.75.19.172}{10.75.19.172:9300}{ml.max\_open\_jobs=10, ml.enabled=true}] 2018-06-11 11:27:23 w-t1K8A Starting datafeed [datafeed-tibco-celect] on node [{w-t1K8A}{w-t1K8AzSmm5UBwmQ7FpWg}{4VahUs23RTGO0js5rRT\_Gw}{10.75.19.172}{10.75.19.172:9300}{ml.max\_open\_jobs=10, ml.enabled=true}] 2018-06-11 11:27:23 w-t1K8A Datafeed started (from: 2018-06-11T15:14:49.576Z to: real-time) 2018-06-11 11:36:55 w-t1K8A Datafeed stopped 2018-06-12 09:32:11 w-t1K8A Starting datafeed [datafeed-tibco-celect] on node [{w-t1K8A}{w-t1K8AzSmm5UBwmQ7FpWg}{4VahUs23RTGO0js5rRT\_Gw}{10.75.19.172}{10.75.19.172:9300}{ml.max\_open\_jobs=10, ml.enabled=true}] 2018-06-12 09:32:11 w-t1K8A Datafeed started (from: 2018-06-11T16:15:00.000Z to: real-time) 2018-06-12 09:32:16 w-t1K8A Datafeed continued in real-time 2018-06-12 09:32:16 w-t1K8A Datafeed lookback completed 2018-06-12 09:33:41 w-t1K8A Datafeed stopped 2018-06-12 09:34:56 w-t1K8A Starting datafeed [datafeed-tibco-celect] on node [{w-t1K8A}{w-t1K8AzSmm5UBwmQ7FpWg}{4VahUs23RTGO0js5rRT\_Gw}{10.75.19.172}{10.75.19.172:9300}{ml.max\_open\_jobs=10, ml.enabled=true}] 2018-06-12 09:34:56 w-t1K8A Datafeed started in real-time 2018-06-12 10:55:00 w-t1K8A Datafeed has been retrieving no data for a while 2018-06-12 14:34:14 w-t1K8A Datafeed stopped ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [June 15, 2018, 12:52am UTC](https://discuss.elastic.co/t/any-ml-job-i-create-runs-for-the-first-time-and-then-stops-and-never-runs/135962/2 "2018-06-15T00:52:26Z")
did you change your `query_delay` value manually? I see in your job config: ```auto "query_delay":"60ms", ``` The above value is likely too small. The default is `60s` ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [June 20, 2018, 10:44am UTC](https://discuss.elastic.co/t/any-ml-job-i-create-runs-for-the-first-time-and-then-stops-and-never-runs/135962/3 "2018-06-20T10:44:37Z")
Spoke to this user in a different communications channel. The root cause of the issue was a subtle bug in the `datafeed` configuration that was introduced in v6.0 and fixed soon after (in 6.0.2 and from 6.1.0 onwards). To workaround the problem, simply set datafeed `frequency` equal to `bucket_span` for count functions ---
**Author:** ![kaushiksoft](https://avatars.discourse-cdn.com/v4/letter/k/49beb7/32.png) [@kaushiksoft](https://discuss.elastic.co/u/kaushiksoft)\ **Post date:** [July 3, 2018, 5:39pm UTC](https://discuss.elastic.co/t/any-ml-job-i-create-runs-for-the-first-time-and-then-stops-and-never-runs/135962/4 "2018-07-03T17:39:20Z")
Yes this work very well. Thank you. Elasticsearch 6.0.0 have this bug. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 30, 2018, 10:21pm UTC](https://discuss.elastic.co/t/any-ml-job-i-create-runs-for-the-first-time-and-then-stops-and-never-runs/135962/5 "2018-10-30T22:21:36Z")