Trigger alert on field exceeding limit per host before it finds other host when its throttled

Created a watcher alert when field exceeds limit, agg by host, throttled for 1 hour and scheduled the trigger for 5 min. The watcher is triggered, found one host which exceeds limit and its being throttled for 1 hour.The watcher again triggered, found other new host then it is also throttled. My question is even its throttled, if it finds a new host it should send an alert of previous found host before it reaches the throttled period.Is there any way to do that?

Hey,

throttling always accounts for the whole watch. If it is not hundreds of thousands of hosts, it might be easier to have an alert per host instead of a grouping one.

--Alex

Sends an alert whenever root exceeds limit per host and throttled for 1 hour
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"metricbeat-*"
],
"types": [],
"body": {
"size": 0,
"query": {
"bool": {
"must": [
{
"term": {
"system.filesystem.mount_point": {
"value": "/"
}
}
},
{
"range": {
"system.filesystem.used.pct": {
"gte": 0.8
}
}
}
]
}
},
"aggs": {
"host": {
"terms": {
"field": "beat.hostname"
},
"aggs": {
"top_hit": {
"top_hits": {
"size": 1
}
}
}
}
}
}
}
}
},
"condition": {
"always": {}
},
"actions": {
"send_email": {
...
...
}
},
"throttle_period_in_millis": 3600000
}

If there is no host then alert contains empty value but it send an email. Do we have any option if its empty ( no host which exceeds limit ) don't send an email?

yes, use a condition that checks for the number of hits in the search response instead of being always true

Thank you, it worked

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.