# Watcher Heartbeat monitor.status query help

**URL:** https://discuss.elastic.co/t/watcher-heartbeat-monitor-status-query-help/139616
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [July 11, 2018, 3:47pm UTC](https://discuss.elastic.co/t/watcher-heartbeat-monitor-status-query-help/139616 "2018-07-11T15:47:16Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![Rob2](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob2](https://discuss.elastic.co/u/Rob2)
#### Post date: [July 11, 2018, 3:47pm UTC](https://discuss.elastic.co/t/watcher-heartbeat-monitor-status-query-help/139616/1 "2018-07-11T15:47:16Z")

</div>

Hi,

I've got a Watch set up with Heartbeat input that sends an alert when any system is not responding to pings (i.e. has monitor.status == down) in the last X minutes.

Here is my watch at the moment:

```
{
  "trigger": {
    "schedule": {
      "interval": "15m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "heartbeat-*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "term": {
                    "monitor.status": {
                      "value": "down"
                    }
                  }
                }
              ],
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "from": "now-15m"
                    }
                  }
                }
              ]
            }
          },
          "aggregations": {
            "by_monitors": {
              "terms": {
                "field": "monitor.host",
                "size": 100,
                "min_doc_count": 1
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          "###@###.##"
        ],
        "subject": "Unresponsive test systems",
        "body": {
          "html": "{{ctx.payload.hits.total}} system(s) not responding to pings:<P>{{#ctx.payload.aggregations.by_monitors.buckets}}{{key}}<BR>{{/ctx.payload.aggregations.by_monitors.buckets}}"
        }
      }
    }
  }
}

```

I'd like to change this so that it only triggers when a given system has monitor.status:down currently AND had monitor.status:up just before that (ex. perhaps now-30m)

How would that be done? Can a transform be used to search a second time using the monitor.host or monitor.id values returned by the above query to find the ones that has monitor.status:up earlier? Any examples or suggestions would be much appreciated.

---

<div class="post-metadata">

### Author: ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [July 13, 2018, 10:59am UTC](https://discuss.elastic.co/t/watcher-heartbeat-monitor-status-query-help/139616/2 "2018-07-13T10:59:45Z")

</div>

hey,

this would be pretty complicated to do in such a generic watch that monitors all your hosts. If you have a watch per host, you could simply call the [Ack Watch API](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/watcher-api-ack-watch.html) yourself to silence a watch after it fires.

Hope this helps!

--Alex

---

<div class="post-metadata">

### Author: ![Rob2](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob2](https://discuss.elastic.co/u/Rob2)
#### Post date: [July 13, 2018, 6:27pm UTC](https://discuss.elastic.co/t/watcher-heartbeat-monitor-status-query-help/139616/3 "2018-07-13T18:27:57Z")

</div>

@spinscale, thanks Alex. Right after firing an email and/or Slack notification, can the "actions" section of a watch be made to acknowledge the action to silence it until after the watch passes? Could you show me what that would look like?

ps. I'm currently completely stuck at the moment where I can't get back to editing my watches:

> [@Unable to get back into Watcher section of Kibana](https://discuss.elastic.co/t/unable-to-get-back-into-watcher-section-of-kibana/139974):
>
> I saved a change to an advanced watch (trying to add a Slack action) and now have a Watcher: Error 400 Bad Request: json argument must contain an actionJson.slack.message.to property error every time I click on Watcher in the Kibana Management page to get back to edit the watch So I'm kind of stuck ... how can I fix this? Is there any way to edit or just delete watches from outside of Kibana?

Is there a way I can edit or remove the problem watch so I can continue working?

---

<div class="post-metadata">

### Author: ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [July 17, 2018, 6:50am UTC](https://discuss.elastic.co/t/watcher-heartbeat-monitor-status-query-help/139616/4 "2018-07-17T06:50:12Z")

</div>

Hey,

I think you may be running into this kibana issue: [https://github.com/elastic/kibana/issues/18532](https://github.com/elastic/kibana/issues/18532)

First, can you share the full watch and your elasticsearch.yml slack configuration? You can do this using the dev tools console and just run the [Get Watch API](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/watcher-api-get-watch.html)

Also, which version of Elasticsearch and Kibana are you running on?

I suppose that it might be easier to use the dev tools for editing watches or adding a `to` parameter in the dev tools, so that your watch UI should be back to working.

I'll try to get a fix in, once all the information is provided.

Thank you!

--Alex

---

<div class="post-metadata">

### Author: ![Rob2](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rob2](https://discuss.elastic.co/u/Rob2)
#### Post date: [July 17, 2018, 10:45pm UTC](https://discuss.elastic.co/t/watcher-heartbeat-monitor-status-query-help/139616/5 "2018-07-17T22:45:21Z")

</div>

Thanks Alex, replied to you on the other post.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 14, 2018, 10:45pm UTC](https://discuss.elastic.co/t/watcher-heartbeat-monitor-status-query-help/139616/6 "2018-08-14T22:45:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
