ML Anomaly Detection - Alert Thresholds

Elastic Stack Kibana
1

Hello,

I recently been building Anomaly Detection jobs and alerts. I was wondering if there's a way to specify a threshold.

Here's my current issue:
Anomaly Detection works by using the historical data as a baseline. If there's a spike/anomaly in the data, the spike would be granted a score of deviation, an anomaly score.

For example:
Lets say, I get an anomaly score of 90, the actual value was 1445 above the baseline of 30

This is a fair score due to it being an anomaly but when it comes to alerting, do I want to know if an anomaly (a deviation from the baseline) occurs all the time? Not really

In some cases, I would only be concerned if the actual value was exceeding 4,000, is this possible to do with the alerts?

2

Then you can create a simple Threshold Alerts / ES| QL alert based on that threshold.

If you want to combine these to data Ml Score + Threshold... not that simple

You could write all the ML Alerts to an index then write a custom rule on top of that data.

1 Like
3

Also look at Custom Rules in the ML job to ignore anomalies that you know are not meaningful.

2 Likes
4

@stephenb @richcollier thanks for the insight!
I ended up using ES|QL and the logic here is that I am grabbing the last hour and the previous hour, then doing math to determine if there was an increase by percentage, in this case, I only care about if its 80% above the baseline

FROM logs-*
| EVAL now = NOW()
| SORT @timestamp
| EVAL key = CASE(@timestamp < (now - 1 hour) AND @timestamp > (now - 2 hour), "Last hour", @timestamp < (now - 2 hour) AND @timestamp > (now - 3 hour), "Previous hour","Other")
| STATS count = COUNT(*) BY key
| EVAL count_last_hour = CASE(key == "Last hour", count), previous_hour = CASE(key == "Previous hour", count)
| STATS last_window_count = SUM(count_last_hour), previous_window_count = SUM(previous_hour)
| EVAL total_percentage = ROUND(ABS((last_window_count::DOUBLE - previous_window_count::DOUBLE) / previous_window_count::DOUBLE), 2) * 100
// Evaluation Threshold
| WHERE previous_window_count > 0 AND total_percentage >= 80

I am still rusty with ES|QL so this query can be optimized but the goal was identify peaks at a percentage , which currently not possible with ML.