Send an email alert after 3 major/critical anomalies for a given time range

[jasgc (Jenniffer Capicio)]

Is there a way for a watcher to generate email alert only after encountering 3 or more major/critical anomalies for a time range (i.e. if it encounters 3 anomalies within 3 months)?

[dadoonet (David Pilato)]

Welcome!

What you can do is to have 2 jobs:

The alerting job you already know. It will write the result in another index.
A new alert job on the alert index.

Building alerts on alerts is normally the way to solve it.

[richcollier]

Certainly, you can construct a Watch to use any logic of your choosing. This, in fact, as described, can be a single Watch (assuming you have an ML job running on the data of interest).

You could, for example, in the input section of the watch, define a search that is over a long period of time:

 "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          ".ml-anomalies-*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "term": {
                    "job_id": "myjobname"
                  }
                },
                {
                  "range": {
                    "timestamp": {
                      "gte": "now-90d"
                    }
                  }
                },
...

Then, in the condition section of the watch, specify how the condition is met:

    "condition": {
      "compare": {
        "ctx.payload.hits.total": {
          "gt": 3
        }
      }
    },

The above would be true if any 3 anomalies are seen in the last 90 days for an ML job named myjobname. Of course, you could add search criteria to only consider anomalies of certain types (bucket, record, influencer, etc.) or of a certain score (either anomaly_score or record_score).

See this blog for more information on scoring: https://www.elastic.co/blog/machine-learning-anomaly-scoring-elasticsearch-how-it-works