Advanced Watcher to send alert of condition has been met for more than 1 hour

[ChrisKelly]

I want to create an advanced Watcher that will only send an alert email out if my conditions have been met more over an hour.

Essentially, I am monitoring specific servers and watching if their CPU exceeds 50%. If it goes above 50% but drops again in the next couple of minutes that's fine but if it exceeds 50% for more than an hour we see problems on those servers.
How can I configure my watcher so it only sends an alert if the CPU goes above 50% for more than an hour?

[GinkoLucas]

Hello,

I think that in the "Input" section, you could get the lowest CPU level received in the hour.

Sets your condition to something like: minCPU > 50

And then adjust your action according to what you want

[ChrisKelly]

In the input isn't the problem, it's the time range, and condition that is the issue.

[GinkoLucas]

The time range must be defined in your input, like this:

{ "range": { "@timestamp": { "gte": "now", "lte": "now - 1h" } }}

And for the condition, it depends on the input request. If you follow my example, you should be able to retrieve the value of the smallest percentage (during the time range you specify, as in my example)

[ChrisKelly]

That will only tell me if the CPU has gone above 50% within the last hour, not if the CPU has been above 50% for the whole hour.

[GinkoLucas]

Not at all. If you get the smallest value during the last hour, and this is greater than 50%, then your CPU was necessarily above 50% during the last hour.

I'm talking about the smallest percentage, not the largest.

[ChrisKelly]

How do I get the smallest percentage from the last hour?

[GinkoLucas]

You can use something like this :

  "aggs" : {
        "mincpu" : {
            "range" : {
                "field" : "@timestamp",
                "ranges" : [
                    { "from" : "now-1h", "to" : "now" }
                ]
            },
            "aggs" : {
                "minpercent" : { "min" : { "field" : "MyPercentage" } }
            }
        }
    }
}

[ChrisKelly]

I have been testing the aggregations above but I can't seem to add the server name so I know which server exceeded the limit. How can I add another aggregation for server name?

[GinkoLucas]

Try Terms aggregation : Terms aggregation | Elasticsearch Guide [8.11] | Elastic

An example in the documentation :

{
  "aggs": {
    "genres": {
      "terms": {
        "field": "genre",
        "order": { "max_play_count": "desc" }
      },
      "aggs": {
        "max_play_count": { "max": { "field": "play_count" } }
      }
    }
  }
}