# ML Anomaly Detection jobs gives very low score for absense of events

**URL:** <https://discuss.elastic.co/t/ml-anomaly-detection-jobs-gives-very-low-score-for-absense-of-events/366963>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [September 23, 2024, 1:48pm UTC](https://discuss.elastic.co/t/ml-anomaly-detection-jobs-gives-very-low-score-for-absense-of-events/366963 "2024-09-23T13:48:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![GlebCA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glebca/32/78044_2.png) [@GlebCA](https://discuss.elastic.co/u/GlebCA)\
**Post date:** [September 23, 2024, 1:48pm UTC](https://discuss.elastic.co/t/ml-anomaly-detection-jobs-gives-very-low-score-for-absense-of-events/366963/1 "2024-09-23T13:48:18Z")

</div>

Hi,  
I like to monitor # of specific logs event and raise an alert when flow is out of boundaries. Initially I was going to use moving average for it but found that [Elastic does not support it and ML anomaly detection is recommended solution](https://discuss.elastic.co/t/today-avg-value-vs-moving-avg-14-days-value-to-trigger-an-alert-based-on-dsl-query/328527)

So I configured Single Metric ML job to detect anomalies. And, unfortunately, when events stop flowing - ML Job did not raise an alert because it assigned Score 4 to this anomaly:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a286b637642afaecbd02fa6b797b142991997b10.png)  
But when IT fixed the problem and events flow resumes - it was an expected spike of buffered events and for this case Score was 92, so the alert was raised.

Why Score for missing events was so low? I need this ML model to raise alerts exactly for missing events or dropping flow.

---

<div class="post-metadata">

**Author:** ![valeriy42](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/valeriy42/32/82758_2.png) [@valeriy42](https://discuss.elastic.co/u/valeriy42)\
**Post date:** [September 24, 2024, 7:37am UTC](https://discuss.elastic.co/t/ml-anomaly-detection-jobs-gives-very-low-score-for-absense-of-events/366963/2 "2024-09-24T07:37:00Z")

</div>

Hello @GlebCA ,

if you are specifically interested in the cases when the event count goes down, you should use `low_count`.

To trigger alerts when the event count drops to 0, it's easiest to [use Kibana alerting and not ML](https://discuss.elastic.co/t/anomaly-on-specific-value-0/290400).

---

<div class="post-metadata">

**Author:** ![GlebCA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glebca/32/78044_2.png) [@GlebCA](https://discuss.elastic.co/u/GlebCA)\
**Post date:** [September 24, 2024, 12:46pm UTC](https://discuss.elastic.co/t/ml-anomaly-detection-jobs-gives-very-low-score-for-absense-of-events/366963/3 "2024-09-24T12:46:15Z")

</div>

Thank you, Valeriy. I already have it in place and it notified IT about the issue 🙂

But I like to have alert when event flow deviates from "normal" without hardcoded values, because this "normal" depends on day of week and # of customers, signed for the service

---

<div class="post-metadata">

**Author:** ![GlebCA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glebca/32/78044_2.png) [@GlebCA](https://discuss.elastic.co/u/GlebCA)\
**Post date:** [October 6, 2024, 10:39pm UTC](https://discuss.elastic.co/t/ml-anomaly-detection-jobs-gives-very-low-score-for-absense-of-events/366963/4 "2024-10-06T22:39:40Z")

</div>

New examples of potential anomalies were collected - ML still assigns too low Severity when values is below expected boundaries:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/4/041394895e0d423440214ffad3561a5fb8ab8893.png)  
Actual value is 10 (TEN) times less then expected but ML job gives this anomaly only Severity 21 ☹

---

<div class="post-metadata">

**Author:** ![valeriy42](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/valeriy42/32/82758_2.png) [@valeriy42](https://discuss.elastic.co/u/valeriy42)\
**Post date:** [October 7, 2024, 11:42am UTC](https://discuss.elastic.co/t/ml-anomaly-detection-jobs-gives-very-low-score-for-absense-of-events/366963/5 "2024-10-07T11:42:34Z")

</div>

Hello @GlebCA ,

To see what is going on, please post the screenshot of the Single Metric Viewer with the model plot activated for the time frame around 2024-10-05 and the configuration of the anomaly detection job.

---

<div class="post-metadata">

**Author:** ![GlebCA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glebca/32/78044_2.png) [@GlebCA](https://discuss.elastic.co/u/GlebCA)\
**Post date:** [October 18, 2024, 2:38am UTC](https://discuss.elastic.co/t/ml-anomaly-detection-jobs-gives-very-low-score-for-absense-of-events/366963/6 "2024-10-18T02:38:05Z")

</div>

Found possible reason - need to use detector [`low_count`](https://www.elastic.co/guide/en/machine-learning/current/ml-count-functions.html#ml-count) instead of 'count'. Going to give it a try
