# Watcher threshold rule does not trigger automatically but manual executes fine

**URL:** <https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [February 20, 2020, 6:09pm UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246 "2020-02-20T18:09:50Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![kaushikrishna](https://avatars.discourse-cdn.com/v4/letter/k/46a35a/32.png) [@kaushikrishna](https://discuss.elastic.co/u/kaushikrishna)\
**Post date:** [February 20, 2020, 6:09pm UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/1 "2020-02-20T18:09:50Z")

</div>

- List item

Hello,

I have just written an advanced watcher rule for threshold alerting (CPU), this rule works fine when I execute the rule using simulate and even sends the alert to slack. However, after saving the rule, it never triggers and the status shows active, I'm not sure if I'm missing a simple step? I have set the threshold to 0.1 % so it should definitely fire, however it does not trigger at all, please see below snippet of manual execute

> Blockquote  
> {  
> "watch\_id": "_inlined_",  
> "node": "7vEAFo9ZSU2DAsr\_9I6j6Q",  
> "state": "executed",  
> "status": {  
> "state": {  
> "active": true,  
> "timestamp": "2020-02-20T18:03:16.041Z"  
> },  
> "last\_checked": "2020-02-20T18:03:16.041Z",  
> "last\_met\_condition": "2020-02-20T18:03:16.041Z",  
> "actions": {  
> "slack\_1": {  
> "ack": {  
> "timestamp": "2020-02-20T18:03:16.041Z",  
> "state": "ackable"  
> },  
> "last\_execution": {  
> "timestamp": "2020-02-20T18:03:16.041Z",  
> "successful": true  
> },  
> "last\_successful\_execution": {  
> "timestamp": "2020-02-20T18:03:16.041Z",  
> "successful": true  
> }  
> }  
> },  
> "execution\_state": "executed",  
> "version": -1  
> },  
> "trigger\_event": {  
> "type": "manual",  
> "triggered\_time": "2020-02-20T18:03:16.041Z",  
> "manual": {  
> "schedule": {  
> "scheduled\_time": "2020-02-20T18:03:16.041Z"  
> }  
> }  
> },  
> "input": {  
> "search": {  
> "request": {  
> "search\_type": "query\_then\_fetch",  
> "indices": [  
> "metricbeat-_"  
> ],  
> "rest\_total\_hits\_as\_int": true,  
> "body": {  
> "size": 0,  
> "query": {  
> "bool": {  
> "must": [  
> {  
> "range": {  
> "@timestamp": {  
> "gte": "{{ctx.trigger.scheduled\_time}}||-10s",  
> "lte": "{{ctx.trigger.scheduled\_time}}",  
> "format": "strict\_date\_optional\_time||epoch\_millis"  
> }  
> }  
> },  
> {  
> "match": {  
> "host.name": "xx.xxx"  
> }  
> },  
> {  
> "match": {  
> "event.dataset": "system.cpu"  
> }  
> }  
> ]  
> }  
> },  
> "aggs": {  
> "metricAgg": {  
> "max": {  
> "field": "system.cpu.total.norm.pct"  
> }  
> }  
> }  
> }  
> }  
> }  
> },  
> "condition": {  
> "script": {  
> "source": "if (ctx.payload.aggregations.metricAgg.value \> params.threshold) { return true; } return false;",  
> "lang": "painless",  
> "params": {  
> "threshold": 0.001  
> }  
> }  
> },  
> "metadata": {  
> "name": "CPU threshold",  
> "xpack": {  
> "type": "json"  
> }  
> },  
> "result": {  
> "execution\_time": "2020-02-20T18:03:16.041Z",  
> "execution\_duration": 476,  
> "input": {  
> "type": "search",  
> "status": "success",  
> "payload": {  
> "\_shards": {  
> "total": 2,  
> "failed": 0,  
> "successful": 2,  
> "skipped": 0  
> },  
> "hits": {  
> "hits": [],  
> "total": 1,  
> "max\_score": null  
> },  
> "took": 7,  
> "timed\_out": false,  
> "aggregations": {  
> "metricAgg": {  
> "value": 0.096  
> }  
> }  
> },  
> "search": {  
> "request": {  
> "search\_type": "query\_then\_fetch",  
> "indices": [  
> "metricbeat-_"  
> ],  
> "rest\_total\_hits\_as\_int": true,  
> "body": {  
> "size": 0,  
> "query": {  
> "bool": {  
> "must": [  
> {  
> "range": {  
> "@timestamp": {  
> "gte": "2020-02-20T18:03:16.041954Z||-10s",  
> "lte": "2020-02-20T18:03:16.041954Z",  
> "format": "strict\_date\_optional\_time||epoch\_millis"  
> }  
> }  
> },  
> {  
> "match": {  
> "host.name": "xx.xxx"  
> }  
> },  
> {  
> "match": {  
> "event.dataset": "system.cpu"  
> }  
> }  
> ]  
> }  
> },  
> "aggs": {  
> "metricAgg": {  
> "max": {  
> "field": "system.cpu.total.norm.pct"  
> }  
> }  
> }  
> }  
> }  
> }  
> },  
> "condition": {  
> "type": "script",  
> "status": "success",  
> "met": true  
> },  
> "transform": {  
> "type": "script",  
> "status": "success",  
> "payload": {  
> "result": 9.6  
> }  
> },  
> "actions": [  
> {  
> "id": "slack\_1",  
> "type": "slack",  
> "status": "success",  
> "slack": {  
> "account": "monitoring",  
> "sent\_messages": [  
> {  
> "status": "success",  
> "message": {  
> "from": "_inlined_",  
> "text": "Watch [CPU threshold] has exceeded the threshold of 8% for node 1. Current CPU utilization is 9.6%."  
> }  
> }  
> ]  
> }  
> }  
> ]  
> },  
> "messages":   
> }

Here's the status of the watch:

> Output  
> curl localhost:9200/\_watcher/watch/0f65255b-8e0d-4aee-b8fc-8d3cf06ed3f4 | json\_pp  
> % Total % Received % Xferd Average Speed Time Time Time Current  
> Dload Upload Total Spent Left Speed  
> 100 1434 100 1434 0 0 175k 0 --:--:-- --:--:-- --:--:-- 200k  
> {  
> "found" : true,  
> "status" : {  
> "version" : 1,  
> "state" : {  
> "active" : true,  
> "timestamp" : "2020-02-19T20:58:01.657Z"  
> },  
> "actions" : {  
> "slack\_1" : {  
> "ack" : {  
> "timestamp" : "2020-02-19T20:58:01.657Z",  
> "state" : "awaits\_successful\_execution"  
> }  
> }  
> }  
> }

--- rest of the output cut off ---

Can you please let me how I can make this rule trigger?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [February 21, 2020, 7:51am UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/2 "2020-02-21T07:51:47Z")

</div>

please share the output of the watcher history entry for this watch in a gist, so we can take a look why the watchdid not trigger

```auto
GET .watcher-history-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "watch_id": "THE_WATCH_ID"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "trigger_event.triggered_time": {
        "order": "desc"
      }
    }
  ]
}

```

hope this helps.

---

<div class="post-metadata">

**Author:** ![kaushikrishna](https://avatars.discourse-cdn.com/v4/letter/k/46a35a/32.png) [@kaushikrishna](https://discuss.elastic.co/u/kaushikrishna)\
**Post date:** [February 23, 2020, 6:42pm UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/3 "2020-02-23T18:42:35Z")

</div>

Hi Alex,

Thanks for helping me out,

The output from .watcher-history-\*/\_search does not contain the watch id for the watch I created. I see only the system watches being present. I wonder if it has something to do with the trigger engine and it not being scheduled at all. What other logs can I provide you to troubleshoot this further?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [February 24, 2020, 8:57am UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/4 "2020-02-24T08:57:18Z")

</div>

Can you provide the watch (the whole API call would be best) that you used to store the watch? I am especially interested in the trigger.

---

<div class="post-metadata">

**Author:** ![kaushikrishna](https://avatars.discourse-cdn.com/v4/letter/k/46a35a/32.png) [@kaushikrishna](https://discuss.elastic.co/u/kaushikrishna)\
**Post date:** [February 24, 2020, 5:26pm UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/5 "2020-02-24T17:26:19Z")

</div>

Here you go, and I saved this watch using the GUI on Kibana. Documentation says the watch is default active when you save it so I didn't use an API call to store the watch, should I instead delete this watch and use an API call to insert it?

{  
"trigger": {  
"schedule": {  
"interval": "30s"  
}  
},  
"input": {  
"search": {  
"request": {  
"search\_type": "query\_then\_fetch",  
"indices": [  
"metricbeat-_"  
],  
"rest\_total\_hits\_as\_int": true,  
"body": {  
"size": 0,  
"query": {  
"bool": {  
"must": [  
{  
"range": {  
"@timestamp": {  
"gte": "{{ctx.trigger.scheduled\_time}}||-30s",  
"lte": "{{ctx.trigger.scheduled\_time}}",  
"format": "strict\_date\_optional\_time||epoch\_millis"  
}  
}  
},  
{  
"match": {  
"agent.hostname": "oam2"  
}  
},  
{  
"match": {  
"event.dataset": "system.cpu"  
}  
}  
]  
}  
},  
"aggs": {  
"metricAgg": {  
"max": {  
"field": "system.cpu.total.norm.pct"  
}  
}  
}  
}  
}  
}  
},  
"condition": {  
"script": {  
"source": "if (ctx.payload.aggregations.metricAgg.value \> params.threshold) { return true; } return false;",  
"lang": "painless",  
"params": {  
"threshold": 0.01  
}  
}  
},  
"actions": {  
"slack\_1": {  
"slack": {  
"message": {  
"text": "Watch [{{ctx.metadata.name}}] has exceeded the threshold of 1% for node 1. Current CPU utilization is {{ctx.payload.result}}%."  
}  
}  
}  
},  
"transform": {  
"script": {  
"source": "HashMap result = new HashMap(); result.result = ctx.payload.aggregations.metricAgg.value_100; return result;",  
"lang": "painless",  
"params": {  
"threshold": 0.01  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![kaushikrishna](https://avatars.discourse-cdn.com/v4/letter/k/46a35a/32.png) [@kaushikrishna](https://discuss.elastic.co/u/kaushikrishna)\
**Post date:** [February 24, 2020, 9:47pm UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/6 "2020-02-24T21:47:30Z")

</div>

Something else I've noticed, JFYI, I created other rules using the simple rule builder and advanced watch on the GUI, and I've inserted a rule using a PUT API call, they all execute manually and return expected results but none of them get triggered automatically and none of them show up in watcher-history.

---

<div class="post-metadata">

**Author:** ![kaushikrishna](https://avatars.discourse-cdn.com/v4/letter/k/46a35a/32.png) [@kaushikrishna](https://discuss.elastic.co/u/kaushikrishna)\
**Post date:** [March 2, 2020, 4:49pm UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/8 "2020-03-02T16:49:04Z")

</div>

On further inspection, it seems that only when slack is configured the automatic triggering does not work. Manual trigger with slack alert still works though.

---

<div class="post-metadata">

**Author:** ![kaushikrishna](https://avatars.discourse-cdn.com/v4/letter/k/46a35a/32.png) [@kaushikrishna](https://discuss.elastic.co/u/kaushikrishna)\
**Post date:** [March 2, 2020, 9:40pm UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/9 "2020-03-02T21:40:05Z")

</div>

Got it working, had to apply slack setting (bin/elasticsearch-keystore) on all three nodes of the Elasticsearch cluster.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 30, 2020, 9:40pm UTC](https://discuss.elastic.co/t/watcher-threshold-rule-does-not-trigger-automatically-but-manual-executes-fine/220246/10 "2020-03-30T21:40:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.