# Watchers Stopped Triggering

**URL:** <https://discuss.elastic.co/t/watchers-stopped-triggering/141212>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [July 23, 2018, 3:42pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212 "2018-07-23T15:42:24Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajrvester](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@ajrvester](https://discuss.elastic.co/u/ajrvester)\
**Post date:** [July 23, 2018, 3:42pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/1 "2018-07-23T15:42:25Z")

</div>

Elastic Cloud Version: 6.2.2

Hello,

a subset of my Watches have stopped firing. There are no reported error messages from the watches themselves, the watch history for the non-triggering watches is empty and the watches fire correctly if we simulate them. This also affects some of the built in watches, as well as the non-advanced watches we've made.

If we open and save them again it seems to resolve the issue, but I'm wondering if there is a known underlying cause for this issue that we can avoid in the future. The only lead I have is we did a cluster update around the time the watches stopped, but I'm not sure from what version.

 ![2%20Months%20Not%20Triggered](https://us1.discourse-cdn.com/elastic/original/3X/d/4/d44937dc5127dd2832e0663d0e43918cda729956.PNG)  
 ![Empty%20Watch%20History](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fa2428ea6e195bd50ce2d06c537705efe3fc7bbf.PNG)

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [July 23, 2018, 3:54pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/2 "2018-07-23T15:54:52Z")

</div>

hey,

can you check if watcher is started by checking the [watcher stats](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/watcher-api-stats.html) and paste the output here?

--Alex

---

<div class="post-metadata">

**Author:** ![ajrvester](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@ajrvester](https://discuss.elastic.co/u/ajrvester)\
**Post date:** [July 23, 2018, 4:10pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/3 "2018-07-23T16:10:42Z")

</div>

Here is the output:

```json
{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "0927d1797c8a646ce539795bf4d18698",
  "manually_stopped": false,
  "stats": [
    {
      "node_id": "pGiZWe_NQi2qIV1b3khC_Q",
      "watcher_state": "started",
      "watch_count": 51,
      "execution_thread_pool": {
        "queue_size": 0,
        "max_size": 10
      }
    }
  ]
}

```

I forgot to note in the original post that most of the watches we have are firing correctly, but the stopped ones all stopped triggering 2 months ago.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [July 23, 2018, 4:45pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/4 "2018-07-23T16:45:57Z")

</div>

hey,

Can you [stop](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/watcher-api-stop.html) and [start](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/watcher-api-start.html) watcher and check with the watcher stats once again if everything is started?

Alternatively before doing that. Could you pick one watch, that does not get triggered currently and just store it again, and see if it gets triggered again?

I assume there is also nothing interesting in the log files? Has this been a multi node cluster at some point in time (I've never seen this so far, so super interested in more information).

Thanks a ton for helping!

--Alex

---

<div class="post-metadata">

**Author:** ![ajrvester](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@ajrvester](https://discuss.elastic.co/u/ajrvester)\
**Post date:** [July 23, 2018, 5:56pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/5 "2018-07-23T17:56:27Z")

</div>

```json
{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "0927d1797c8a646ce539795bf4d18698",
  "manually_stopped": false,
  "stats": [
    {
      "node_id": "pGiZWe_NQi2qIV1b3khC_Q",
      "watcher_state": "started",
      "watch_count": 66,
      "execution_thread_pool": {
        "queue_size": 0,
        "max_size": 10
      }
    }
  ]
}

```

Stop and start seems to have bumped the watch count up, and the previously inactive watchers are starting to fire again.

I tried storing 2 watches earlier and it seems to have made them start running again. Pretty certain this has only been a single node cluster since we launched it.

In the log files there are some watch execution failures for the built in watches. Not sure how to format this nicely, sorry:

```auto
[2018-07-23T17:52:24,980][ERROR][org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform] failed to execute [script] transform for [DrkweI7bRXCjKJXMoXRdhQ_elasticsearch_cluster_status_6c0232bc-b1be-49d4-92f8-cc03f77a6d53-2018-07-23T17:52:24.973Z] org.elasticsearch.script.ScriptException: runtime error at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:101) ~[?:?] at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ctx.vars.is_new = ctx.vars.fails_check && !ctx.vars.not_resolved;ctx.vars.is_resolve ...:1070) ~[?:?] at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:105) ~[?:?] at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:69) ~[x-pack-watcher-6.2.2.jar:6.2.2] at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:53) ~[x-pack-watcher-6.2.2.jar:6.2.2] at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) ~[x-pack-watcher-6.2.2.jar:6.2.2] at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:481) ~[x-pack-watcher-6.2.2.jar:6.2.2] at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:322) ~[x-pack-watcher-6.2.2.jar:6.2.2] at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$7(ExecutionService.java:426) ~[x-pack-watcher-6.2.2.jar:6.2.2] at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:580) [x-pack-watcher-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.2.jar:6.2.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_144] at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_144] at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ctx.vars.is_new = ctx.vars.fails_check && !ctx.vars.not_resolved;ctx.vars.is_resolve ...:347) ~[?:?] ... 12 more

```

Thanks for the help!

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [July 24, 2018, 7:24am UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/6 "2018-07-24T07:24:22Z")

</div>

Hey,

Glad everything is working again. I'll take a closer at the source the next days to see if anything stands out what could have caused this.

do you happen to have a watch history entry for a watch execution of the with the id `DrkweI7bRXCjKJXMoXRdhQ_elasticsearch_cluster_status` around the logging timestamp?

--Alex

---

<div class="post-metadata">

**Author:** ![ajrvester](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@ajrvester](https://discuss.elastic.co/u/ajrvester)\
**Post date:** [July 24, 2018, 1:28pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/7 "2018-07-24T13:28:33Z")

</div>

Here is a [pastebin](https://pastebin.com/yZFBHnGa) with the watcher result.

A co-worker last night noted the watcher stoppage might have coincided with a configuration update we had made about 2 months ago. Last night we made a similar configuration change on a monitoring cluster we have and it seems to have resulted in a similar watcher stoppage. Fixed it with the same stop/start commands.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [July 24, 2018, 1:44pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/8 "2018-07-24T13:44:39Z")

</div>

can you provide some more information what you did? I am curious what may have caused this, and what we can do to prevent this in the future or come up with better error messages, if it is possible to prevent his from happening to others.

---

<div class="post-metadata">

**Author:** ![ajrvester](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@ajrvester](https://discuss.elastic.co/u/ajrvester)\
**Post date:** [July 24, 2018, 2:04pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/9 "2018-07-24T14:04:50Z")

</div>

I went into the Elastic Cloud console and changed the `elasticsearch.yml` to add a PagerDuty integration. We just had a Slack integration configured beforehand.

The "grow and shrink" configuration change seems to have executed without errors. I didn't see anything weird in the logs while the configuration change was running.

This is what our configuration looks like now on both clusters:

```auto
xpack.notification.slack:
  account:
    monitoring:
      url: [URL]
xpack.notification.pagerduty:
  account:
    [account-name]:
      service_api_key: [api-key]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 21, 2018, 2:05pm UTC](https://discuss.elastic.co/t/watchers-stopped-triggering/141212/10 "2018-08-21T14:05:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.