# Machine Learning - Watcher - Not Firing **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning, elastic-stack-alerting\ **Created:** [September 28, 2018, 3:26pm UTC](https://discuss.elastic.co/t/machine-learning-watcher-not-firing/150361 "2018-09-28T15:26:17Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Post date:** [September 28, 2018, 3:26pm UTC](https://discuss.elastic.co/t/machine-learning-watcher-not-firing/150361/1 "2018-09-28T15:26:17Z")
I've set up a ML job that tracks all of the servers that we have deployed metricbeat on by using the Detector low\_count thats then split by beat.name so that each server is tracked. I've setup a Watcher that matches the following condition: When count(), overall all documents, is below 1 for the last 1 minute an email/log is generated. The idea behind this watcher is to let us know that a Metricbeat has stopped reporting completely and that there may be an issue with a server due to the fact that its not sending any documents. This works great with the 1 minute interval but when you set, for the last, to 5 minutes it never triggers when we turn off Metricbeat on a specific server. Is this due to the fact that it is seeing Metricbeat data coming in from other servers over the 5 minute span so they are in essence overriding the fact that MB is not reporting on one server. Also, out aggregation interval and bucket span are set to 1 minute if that makes a difference. Appreciate the insight. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [September 28, 2018, 3:38pm UTC](https://discuss.elastic.co/t/machine-learning-watcher-not-firing/150361/2 "2018-09-28T15:38:03Z")
hey, how about changing your watch a little bit (not sure if the watcher UI is capable of that), so that you run a terms aggregation on the hostnames for the last minute, and another terms agg on hostnames for the last 5 minutes, then compare those two in the condition and check if any are missing? --Alex ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 29, 2018, 6:00pm UTC](https://discuss.elastic.co/t/machine-learning-watcher-not-firing/150361/3 "2018-09-29T18:00:39Z")
I'd be curious as to how you set up the watch itself (the actual JSON) but one thing to keep in mind is that ML jobs write results with a timestamp equal to the leading edge of the `bucket_span`. So if your `bucket_span` is 5m, and if an anomaly occurs in the interval of 11:00-11:05, the anomaly record is written with a timestamp=11:00. And, since there is also a slight delay in the ML job due to the `query_delay` parameter, then this results document which has a timestamp=11:00 is really written with a clock time of perhaps a little past 11:06. So, if your watch isn't accounting for this, you will "miss" seeing the anomaly document. See more info: [https://www.elastic.co/blog/alerting-on-machine-learning-jobs-in-elasticsearch-v55](https://www.elastic.co/blog/alerting-on-machine-learning-jobs-in-elasticsearch-v55) ---
**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Post date:** [October 2, 2018, 2:57pm UTC](https://discuss.elastic.co/t/machine-learning-watcher-not-firing/150361/4 "2018-10-02T14:57:34Z")
So I've ben able to get some watchers to fire at a more consistent rate so thats a step in the right direction. What I'm learing however is that if you use one or just a few boxes to turn metricbeat on and off the ML program pick up on that and they don't produce critical errors to easily sort the servers by in the alerting dashboard. This seems a little odd with the ML metrics due to the fact that the server is not reporting at all, which should be a critical error, yet ML only produces the lowest level warning of, warning. I do get how testing this multiple times effects ML but it still seems odd to me that critical warnings are not produced when metricbeat stops reporting. I'll try to post what I have once I fine tune it all. ---
**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Post date:** [October 2, 2018, 7:14pm UTC](https://discuss.elastic.co/t/machine-learning-watcher-not-firing/150361/5 "2018-10-02T19:14:18Z")
Stumbled across a really good reference to work off of. Hope this help anyone thats looking to do something similar. [https://github.com/elastic/examples/blob/master/Alerting/Sample%20Watches/system\_fails\_to\_provide\_data/watch.json](https://github.com/elastic/examples/blob/master/Alerting/Sample%20Watches/system_fails_to_provide_data/watch.json) ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [October 2, 2018, 7:25pm UTC](https://discuss.elastic.co/t/machine-learning-watcher-not-firing/150361/6 "2018-10-02T19:25:37Z")
Yes, the more you test, the more you reinforce to ML that the behavior of being "off" is normal and it will indeed be naturally scored less - that's not surprising. And yes, if you're just looking for a binary on/off condition, then ML is overkill and you can accomplish everything with a pretty straightforward watch. ML comes into play when you're trying to detect things that aren't so easy to define with a rule/threshold, like the anomaly near the right hand side here: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/f/0fb232ef513fbb88c1fba648e709360b1b1e5510.jpeg) Cheers ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 30, 2018, 7:25pm UTC](https://discuss.elastic.co/t/machine-learning-watcher-not-firing/150361/7 "2018-10-30T19:25:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.