Watcher when a server goes down and a followup if it's been down for 24 hours?

Elastic Stack Kibana
1

Hello,

I have a number of servers being pinged by heartbeat. What I'm trying to figure out how to do is:

  • "Watcher 1" run every minute - When a server has been consistently down for a minute (no monitor.status = up for any documents for one particular address in heartbeat for the past 60 seconds of documents), send an email notification, then trigger a "watcher" to evaluate again every 24 hours.
    - "Watcher 2" - At the 24 hour mark, if the server has still been consistently down (no monitor.status = up for any document for that one address pinged in any of the heartbeat documents returned for that server for the past 24 hours), trigger another email.
    - During this "wait 24 hour" period, suspend "Watcher 1" running every minute on the server

Is something like this doable with the watcher functionality? I've seen workflows posted before about a simple downtime notification, but not sure if the rest of this can be performed.

2

@devbot You definitely can do something like this with Watcher. Unfortunately I don't have a working example to share directly with you, but there's a good corpus of information here if you haven't seen it already.

You'll also need to configure email settings in Watcher to do so.