Recently we experienced having a slew of alarms/notifications going to our slack channel without our instances actually going down. Uptime monitors also show they were up in the time we received the notifications. Is this a common error? Should I edit the rules to be less sensitive? What elements can cause alerts to be sent when monitors aren't actually going down?
Hi Wilfredo. Monitors triggering alerts when they are marked as up is unusual, because the presence of an explicit down monitor is required to trigger the alert.
We are discussing this internally, and want to better understand how this could happen. Could you give us some general information on your set up to think through this: Which version of Kibana are you on? Which version of Heartbeat or Elastic Agent are you running? Are you using the beta Monitor Management function in Uptime, and if so are you running against public or private locations? Are you using lightweight monitors, browser monitors, or both? Did the alerts trigger for all your monitors, or just one type of monitor (browser monitors for example)?
One thing to note is that for the purposes of alerting, a monitor is considered down from the time it runs through to when the next scheduled monitor is set to run. Let's say you have a monitor that runs every hour, and it went down at 12:00:00. Since the monitor is set to run every hour, that monitor is considered down for the purposes of alerting from 12:00:00 - 13:00:000. If you configure a rule and it runs at let's say, 12:59:00, the monitor that ran 59 minutes prior is still considered down, and will trigger an alert.
We don't have any further suggestions at this time, such as editing the rule to be less sensitive. Sensitivity in a rule can sometimes cause alerts to be resolved while the monitor is still down, but should not have an impact on alert triggering when the monitors are marked as up.
As I mentioned this is quite unusual, so we're definitely keen to get to the bottom of what's causing this issue.
1 Like
Once we made the alerts less sensitive, the massive alerting got under control. Just a few that might be having DNS issues. Thanks for the reply!
All email and communication should be considered private and protected personal and shall be secured under all applicable laws. Any recipient, who has access to any contained information, regardless of purpose or medium is so advised, bound and obligated to handle as such.
This topic was automatically closed 24 days after the last reply. New replies are no longer allowed.