Trigger alert on field exceeding limit per host before it finds other host when its throttled

alerting

#1

Created a watcher alert when field exceeds limit, agg by host, throttled for 1 hour and scheduled the trigger for 5 min. The watcher is triggered, found one host which exceeds limit and its being throttled for 1 hour.The watcher again triggered, found other new host then it is also throttled. My question is even its throttled, if it finds a new host it should send an alert of previous found host before it reaches the throttled period.Is there any way to do that?


(Alexander Reelsen) #2

Hey,

throttling always accounts for the whole watch. If it is not hundreds of thousands of hosts, it might be easier to have an alert per host instead of a grouping one.

--Alex


#3

Sends an alert whenever root exceeds limit per host and throttled for 1 hour
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"metricbeat-*"
],
"types": [],
"body": {
"size": 0,
"query": {
"bool": {
"must": [
{
"term": {
"system.filesystem.mount_point": {
"value": "/"
}
}
},
{
"range": {
"system.filesystem.used.pct": {
"gte": 0.8
}
}
}
]
}
},
"aggs": {
"host": {
"terms": {
"field": "beat.hostname"
},
"aggs": {
"top_hit": {
"top_hits": {
"size": 1
}
}
}
}
}
}
}
}
},
"condition": {
"always": {}
},
"actions": {
"send_email": {
...
...
}
},
"throttle_period_in_millis": 3600000
}

If there is no host then alert contains empty value but it send an email. Do we have any option if its empty ( no host which exceeds limit ) don't send an email?


(Alexander Reelsen) #4

yes, use a condition that checks for the number of hits in the search response instead of being always true


#5

Thank you, it worked


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.