Problems with SCORE on Anomaly Detection JOB

[Rossana (Raquel Acero)]

hi!

I have this behavior

The typical value was 18.3 and the Actual vale was 0.

The anomaly score is too low and I should had received an anomaly alert but the score was wrong, so the alert was not fired.

I also saw a cross mark and I dont know what that means?

Please help me to understand this behavior! and how to tune it!

[valeriy42 (Valeriy Khakhutskyy)]

Hello @Rossana ,

The cross mark means that the multi-bucket behavior is mainly responsible for the anomaly score. Please, read the blog post on Explaining anomalies detected by Elastic machine learning.

From the screenshot snippet, it's unclear what was the previous behavior in the data. However, the anomalies are preceded by a long period of 0 values, which were typical, and in the interval of interest, the difference between the actual value of 0 and the lower bound of the confidence interval apparently is not that large to motivate a higher score.

• You can the *_low anomaly detection function if you are interested in drops
• Consider setting an alert to a specific rule, if you are interested in being notified, e.g. if you receive 0 values after 7 am.
• You can also set the anomaly score threshold to a lower value like 17 trigger alarm for such behavior.

[Rossana (Raquel Acero)]

Hi,

Thanks for your anwers. I read the link but still confused by this topic.

I had a production problem so I need to prevent this happen again because no alerts was shot. The same day we had a lot of errors as you can see in the next picture:

This image considere the 11 days before an the value was 42x higher and the severity was <1. WHY?

This is the same event but here there is only two days from now.

I understand that was a multibucket impact (still a bit confusing) but what happen with the single bucket impact? or the value actual and the typical? and why the score was <1 (what means <1)?

I feel lose in this topic!

[valeriy42 (Valeriy Khakhutskyy)]

Hello @Rossana ,

I am sorry you feel confused about the scores. Here are some pointers to help you:

1. As a rule of thumb, an anomaly detector needs about 3 weeks of data to build a probabilistic model that describes the data. In your case, it appears that the data ingest started on 2023-07-11 and the anomalous behavior is observed only 6 days later. Therefore, you need to let the job run for a bit longer before trying to understand the "usual" score numbers that the anomaly detector would assign. Before seeing enough evidence (data), the anomaly detector would be reluctant to assign high anomaly scores, since the "typical" value is derived from an insufficient number of observations.
2. Anomaly detectors make sense on complex data with multiple seasonalities (e.g. different behavior over hour of the day, day of the week, month, etc.), trends, and so on. If you have data where you expect ~0 most of the time and want to be alerted when you get anything >100, then a simple alert rule may be more helpful.
3. There are many resources available online that dive deep into how anomalies are identified and scored.

[Rossana (Raquel Acero)]

Hi,

Thanks but I have a lot of data... More than 60 days:

As you can see in the image before my data has differents behaviors.
I receive errors evey day but I only want and alert when the amount of error are anomalous.

This happen last night.

Actual: 186
Typical: 3.56
Still have anomaly score <1.
Here there is no multi bucket impact

1-. Why this happen?
2-. Why <1 and why is the meaning of this score?

[valeriy42 (Valeriy Khakhutskyy)]

Hi,

well, that's interesting... Can you please let us know what version of ES you are using? On the first screenshot, there is an annotation marked as [1] around 2023-07-02. Can you please tell me what this annotation is? It seems that before this annotation, the model score behaved as you would expect.

Also for the last spike can you please put the screenshot of anomaly score explanations (available if you click on the arrow in the anomaly table)?

Thank you.

[Rossana (Raquel Acero)]

Hi,

Here the information you ask for...

1-. Cluster Version:

2-. Annotation:

I dont know what it is? but if I click it, this is what it shows...

3-. Anomaly score explanation:

Thanks.

[Rossana (Raquel Acero)]

@valeriy42 can you help me?

[valeriy42 (Valeriy Khakhutskyy)]

Hello @Rossana ,

Sorry for the delay. Removing all seasonalities at [1] doesn't seem right. I'll reach out to colleagues to investigate it further.

Does your data contain buckets with no entries? Please check if using high_non_zero_count instead of high_count leads to the same results.

[Rossana (Raquel Acero)]

Hi,

Did you receive an answer form your collagues?

any news!

[valeriy42 (Valeriy Khakhutskyy)]

Hello,

here are the results of our analysis:

1. You base how high you think the anomaly score should be on how different the value is from "typical" (this is the median prediction). Often the value is very different from typical, so by itself this is not necessarily a good indicator that something is anomalous because the data has a long positive tail.

2. Our score is trying to say whether these events are historically unusual. Looking at the history of the metric, this event does not look particularly unusual. However, one option you have is that if you continue to have problems, you can consider reverting to the snapshot and excluding the problem periods.

3. Perhaps this metric is not a good indicator of problems. Do you really have problems like this all the time (based on the screenshot)? Perhaps the metric you are analyzing for anomalies does not correlate well with problems in your system. Some metrics are better at detecting problems, like KPIs, and some can be useful for root cause and are too unpredictable to perform reliable anomaly detection for alerting.

4. If you like, you can create an alert based on what differs from typical, if you think this better detects the events you care about. A watch on the anomaly detection results would allow you to do this.

5. There's nothing that says you have to alert at a certain level, so you can create a rule that alerts at a lower score if that doesn't give you too many false positives.

6. The blow-up in bounds looks a bit like [ML] Improve robustness to outliers after detecting changes in time series by tveasey · Pull Request #2280 · elastic/ml-cpp · GitHub, this isn't in your version and might improve the behavior after reinitialization. However, I would still say that all the above observations apply.