I am trying to find out a way to do alerting on downed hosts with watcher though I am running into an issue trying to figure out how to handle throttling the alerts. Once a given host is down I only want it to send one email alert then throttle for 24 hours. But I still want it to alert if a new host goes down meaning I can't really use the throttle feature.
One solution could be to make a watcher for each host but I don't know what the scalability/performance impact of that would be.
Has anyone else ran into a similar situation with alerting using watcher?