# Problems with SCORE on Anomaly Detection JOB **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-machine-learning\ **Created:** [July 17, 2023, 6:12pm UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600 "2023-07-17T18:12:28Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Rossana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rossana/32/94216_2.png) [@Rossana](https://discuss.elastic.co/u/Rossana)\ **Post date:** [July 17, 2023, 6:12pm UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/1 "2023-07-17T18:12:28Z")
hi! I have this behavior ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/1/71927cd5abcd9adbf5dbd0512840585cc7f80b39.png) The typical value was 18.3 and the Actual vale was 0. ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/a/2aa602612968b02e0c3f0074d1758f642c64eb1d.png) The anomaly score **is too low** and I should had **received** an **anomaly alert** but the **score was wrong** , so the **alert was not fired**. I also saw a **cross mark** and I dont know what that means? Please help me to understand this behavior! and how to tune it! ---
**Author:** ![valeriy42](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/valeriy42/32/82758_2.png) [@valeriy42](https://discuss.elastic.co/u/valeriy42)\ **Post date:** [July 18, 2023, 12:09pm UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/2 "2023-07-18T12:09:16Z")
Hello @Rossana , The cross mark means that the multi-bucket behavior is mainly responsible for the anomaly score. Please, read the blog post on [Explaining anomalies detected by Elastic machine learning](https://www.elastic.co/blog/explaining-anomalies-detected-by-elastic-machine-learning). From the screenshot snippet, it's unclear what was the previous behavior in the data. However, the anomalies are preceded by a long period of 0 values, which were typical, and in the interval of interest, the difference between the actual value of 0 and the lower bound of the confidence interval apparently is not that large to motivate a higher score. - You can the `*_low` anomaly detection function if you are interested in drops - Consider setting an alert to a specific rule, if you are interested in being notified, e.g. if you receive 0 values after 7 am. - You can also set the anomaly score threshold to a lower value like 17 trigger alarm for such behavior. ---
**Author:** ![Rossana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rossana/32/94216_2.png) [@Rossana](https://discuss.elastic.co/u/Rossana)\ **Post date:** [July 18, 2023, 3:25pm UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/3 "2023-07-18T15:25:49Z")
Hi, Thanks for your anwers. I read the link but still confused by this topic. I had a production problem so I need to prevent this happen again because no alerts was shot. The same day we had a lot of errors as you can see in the next picture: ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/5/75cf421231b9d1215710500ac3543d107504b263.png) This image considere the **11 days** before an the value was **42x higher** and the severity was **\<1**. WHY? This is the same event but here there is only two days from now. ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/8/388adc989fb50a4022b6f58038526eb660f1d40d.png) I understand that was a multibucket impact (still a bit confusing) but what happen with the single bucket impact? or the value actual and the typical? and why the score was **\<1** (what means \<1)? I feel lose in this topic! ---
**Author:** ![valeriy42](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/valeriy42/32/82758_2.png) [@valeriy42](https://discuss.elastic.co/u/valeriy42)\ **Post date:** [July 19, 2023, 8:05am UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/4 "2023-07-19T08:05:04Z")
Hello @Rossana , I am sorry you feel confused about the scores. Here are some pointers to help you: 1. As a rule of thumb, an anomaly detector needs about 3 weeks of data to build a probabilistic model that describes the data. In your case, it appears that the data ingest started on 2023-07-11 and the anomalous behavior is observed only 6 days later. Therefore, you need to let the job run for a bit longer before trying to understand the "usual" score numbers that the anomaly detector would assign. Before seeing enough evidence (data), the anomaly detector would be reluctant to assign high anomaly scores, since the "typical" value is derived from an insufficient number of observations. 2. Anomaly detectors make sense on complex data with multiple seasonalities (e.g. different behavior over hour of the day, day of the week, month, etc.), trends, and so on. If you have data where you expect ~0 most of the time and want to be alerted when you get anything \>100, then a simple alert rule may be more helpful. 3. There are [many resources](https://discuss.elastic.co/t/anomaly-detection/309098/2) available online that dive deep into how anomalies are identified and scored. ---
**Author:** ![Rossana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rossana/32/94216_2.png) [@Rossana](https://discuss.elastic.co/u/Rossana)\ **Post date:** [July 19, 2023, 1:44pm UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/5 "2023-07-19T13:44:07Z")
Hi, Thanks but I have a lot of data... More than 60 days: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9c4f33d054dc4f846b03e4dd2f2b2c8408ad91f7.png) As you can see in the image before my data has differents behaviors. I receive errors evey day but I only want and alert when the amount of error are anomalous. This happen last night. ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/8/78bf828d5860a73e273cd56e13e284cffd172fb3.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9cec5ae1e46feebe9af8c2c19f77dce4258a8d43.png) **Actual: 186** **Typical: 3.56** **Still have anomaly score \<1. 😩 😫** Here there is no multi bucket impact 😪 1-. Why this happen? 2-. Why **\<1** and why is the meaning of this score? 😖 ---
**Author:** ![valeriy42](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/valeriy42/32/82758_2.png) [@valeriy42](https://discuss.elastic.co/u/valeriy42)\ **Post date:** [July 20, 2023, 7:01am UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/6 "2023-07-20T07:01:29Z")
Hi, well, that's interesting... Can you please let us know what version of ES you are using? On the first screenshot, there is an annotation marked as [1] around 2023-07-02. Can you please tell me what this annotation is? It seems that before this annotation, the model score behaved as you would expect. Also for the last spike can you please put the screenshot of anomaly score explanations (available if you click on the arrow in the anomaly table)? Thank you. ---
**Author:** ![Rossana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rossana/32/94216_2.png) [@Rossana](https://discuss.elastic.co/u/Rossana)\ **Post date:** [July 20, 2023, 6:17pm UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/7 "2023-07-20T18:17:18Z")
Hi, Here the information you ask for... **1-. Cluster Version:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5d87de9c82c8a27d8326bad50ebdbb08d51344ba.png) **2-. Annotation:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5e523e1b81c233ad0f8f505cb9e30d114126ba14.png) I dont know what it is? but if I click it, this is what it shows... ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/2/d241e08263b5a695447021e8e5819efd905f6c12.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b67811165ab0bb984439e3cebb7e423728cefeec.png) **3-. Anomaly score explanation:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/e/eef0147da9ad564d72326a9de930ad409a079180.png) Thanks. ---
**Author:** ![Rossana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rossana/32/94216_2.png) [@Rossana](https://discuss.elastic.co/u/Rossana)\ **Post date:** [July 25, 2023, 4:30pm UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/8 "2023-07-25T16:30:23Z")
@valeriy42 can you help me? ---
**Author:** ![valeriy42](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/valeriy42/32/82758_2.png) [@valeriy42](https://discuss.elastic.co/u/valeriy42)\ **Post date:** [July 26, 2023, 10:54am UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/9 "2023-07-26T10:54:05Z")
Hello @Rossana , Sorry for the delay. Removing all seasonalities at [1] doesn't seem right. I'll reach out to colleagues to investigate it further. Does your data contain buckets with no entries? Please check if using `high_non_zero_count` instead of `high_count` leads to the same results. ---
**Author:** ![Rossana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rossana/32/94216_2.png) [@Rossana](https://discuss.elastic.co/u/Rossana)\ **Post date:** [August 17, 2023, 12:21am UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/10 "2023-08-17T00:21:49Z")
Hi, Did you receive an answer form your collagues? any news! ---
**Author:** ![valeriy42](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/valeriy42/32/82758_2.png) [@valeriy42](https://discuss.elastic.co/u/valeriy42)\ **Post date:** [August 17, 2023, 8:32am UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/11 "2023-08-17T08:32:38Z")
Hello, here are the results of our analysis: 1. You base how high you think the anomaly score should be on how different the value is from "typical" (this is the median prediction). Often the value is very different from typical, so by itself this is not necessarily a good indicator that something is anomalous because the data has a long positive tail. 2. Our score is trying to say whether these events are historically unusual. Looking at the history of the metric, this event does not look particularly unusual. However, one option you have is that if you continue to have problems, you can consider reverting to the snapshot and excluding the problem periods. 3. Perhaps this metric is not a good indicator of problems. Do you really have problems like this all the time (based on the screenshot)? Perhaps the metric you are analyzing for anomalies does not correlate well with problems in your system. Some metrics are better at detecting problems, like KPIs, and some can be useful for root cause and are too unpredictable to perform reliable anomaly detection for alerting. 4. If you like, you can create an alert based on what differs from typical, if you think this better detects the events you care about. A watch on the anomaly detection results would allow you to do this. 5. There's nothing that says you have to alert at a certain level, so you can create a rule that alerts at a lower score if that doesn't give you too many false positives. 6. The blow-up in bounds looks a bit like [[ML] Improve robustness to outliers after detecting changes in time series by tveasey · Pull Request #2280 · elastic/ml-cpp · GitHub](https://github.com/elastic/ml-cpp/pull/2280), this isn't in your version and might improve the behavior after reinitialization. However, I would still say that all the above observations apply. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 14, 2023, 8:33am UTC](https://discuss.elastic.co/t/problems-with-score-on-anomaly-detection-job/338600/12 "2023-09-14T08:33:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.