# Watch is stuck at error, always needs a manual restart **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-alerting\ **Created:** [September 24, 2020, 8:59am UTC](https://discuss.elastic.co/t/watch-is-stuck-at-error-always-needs-a-manual-restart/249778 "2020-09-24T08:59:12Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [September 24, 2020, 8:59am UTC](https://discuss.elastic.co/t/watch-is-stuck-at-error-always-needs-a-manual-restart/249778/1 "2020-09-24T08:59:12Z")
Hi, I'm using Elastic Stack 7.8. I've enable a watcher. Every night it shows error: connection timeout for webhook action. And the action status in But only when I edit the watch(I'm not actually changing any thing, really) and save it again in the morning, it starts working normally. This error: ``` "actions": [ { "id": "webhook_1", "type": "webhook", "status": "failure", "error": { "root_cause": [ { "type": "connect_timeout_exception", "reason": "Connect to 1.2.3.4:8081 [/1.2.3.4] failed: connect timed out" } ], "type": "connect_timeout_exception", "reason": "Connect to 1.2.3.4:8081 [/1.2.3.4] failed: connect timed out", "caused_by": { "type": "socket_timeout_exception", "reason": "connect timed out" } } } ] }, ``` Followed all the other errors: ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/c/ac86be2907284c37e438d90ace3d10e09253f2a4.png) Why does it get stuck when watch encounters an error? Can't it just move on and try again later? If it requires a restart then is there a restart command, like there was in 6.8 : `POST _xpack/watcher/_restart`? I'm not looking for solution for Connection timeout, I just want to restart watch even after a connection error and try again after it is triggered again. Thanks. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [September 24, 2020, 9:46am UTC](https://discuss.elastic.co/t/watch-is-stuck-at-error-always-needs-a-manual-restart/249778/2 "2020-09-24T09:46:00Z")
Hey, when the watch is stuck and cannot be executed, can you run the [Watcher Stats API](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/watcher-api-stats.html) with the `emit_stacktraces` parameter and share the output in a gist? Thanks! --Alex ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [September 26, 2020, 6:30pm UTC](https://discuss.elastic.co/t/watch-is-stuck-at-error-always-needs-a-manual-restart/249778/3 "2020-09-26T18:30:09Z")
Hi, Alex. Thanks for replying. This watcher works well when I click on 'Send Request' button, while creating a test watch, and check its output. But only when it is left to execute on its own it throws: connect\_timeout\_exception. This is the output of `GET _watcher/stats?emit_stacktraces=true`: ``` { "_nodes" : { "total" : 8, "successful" : 8, "failed" : 0 }, "cluster_name" : "abc", "manually_stopped" : false, "stats" : [ { "node_id" : "lyELuRG0QAS64C93O-oqRg", "watcher_state" : "started", "watch_count" : 0, "execution_thread_pool" : { "queue_size" : 0, "max_size" : 1 } }, { "node_id" : "eEJyFdOwT3-MCKTGVx3p4w", "watcher_state" : "started", "watch_count" : 2, "execution_thread_pool" : { "queue_size" : 0, "max_size" : 40 } }, { "node_id" : "O3ezUXu2Q8qUq62scF11_g", "watcher_state" : "started", "watch_count" : 0, "execution_thread_pool" : { "queue_size" : 0, "max_size" : 0 } }, { "node_id" : "y6wGcIPQRHmhePoYtxI01Q", "watcher_state" : "started", "watch_count" : 0, "execution_thread_pool" : { "queue_size" : 0, "max_size" : 0 } }, { "node_id" : "57shI5qnS86BfgMSKaSsYg", "watcher_state" : "started", "watch_count" : 0, "execution_thread_pool" : { "queue_size" : 0, "max_size" : 40 } }, { "node_id" : "qR_AGOwJRNuBxsqPOWuRrA", "watcher_state" : "started", "watch_count" : 8, "execution_thread_pool" : { "queue_size" : 0, "max_size" : 40 } }, { "node_id" : "u9zQ0k-gS72zDbftee7OIQ", "watcher_state" : "started", "watch_count" : 0, "execution_thread_pool" : { "queue_size" : 0, "max_size" : 0 } }, { "node_id" : "wAP1CsuERLOQdECfH4y3ew", "watcher_state" : "started", "watch_count" : 0, "execution_thread_pool" : { "queue_size" : 0, "max_size" : 0 } } ] } ``` The watcher is working on qR\_AGOwJRNuBxsqPOWuRrA node and when i checked node/stats I saw the following output: ``` "failures" : [ { "type" : "failed_node_exception", "reason" : "Failed node [qR_AGOwJRNuBxsqPOWuRrA]", "node_id" : "qR_AGOwJRNuBxsqPOWuRrA", "caused_by" : { "type" : "translog_exception", "reason" : "Unable to get the earliest last modified time for the transaction log", "index_uuid" : "Gd5TSGq7Qoaut5hrpNocJw", "shard" : "0", "index" : ".kibana_1" } } ] ``` } Could that be the reason? .kibana\_1 index has 2 shards and out of them 1 has missing translog files. How do I fix that? Thanks. ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [September 26, 2020, 8:00pm UTC](https://discuss.elastic.co/t/watch-is-stuck-at-error-always-needs-a-manual-restart/249778/4 "2020-09-26T20:00:56Z")
I read @s1monw's response here: [Cannot recover index because of missing tanslog files](https://discuss.elastic.co/t/cannot-recover-index-because-of-missing-tanslog-files/38336/6) but I didn't quite understand how renaming a file would help me. Also my translog file is already named: translog-11.tlog and translog.ckp. Any help would be appreciated. Thanks! ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [September 28, 2020, 8:13am UTC](https://discuss.elastic.co/t/watch-is-stuck-at-error-always-needs-a-manual-restart/249778/5 "2020-09-28T08:13:11Z")
Also, that index is about kibana. Watcher does not store any information in there. What we can infer from the response that you pasted, is that at the time you executed that request, there are no stuck watches. Can you share the last run via the watcher history ```auto GET .watcher-history-*/_search { "query": { "bool": { "filter": [ { "term": { "watch_id": "my_watch" } } ] } }, "sort": [ { "trigger_event.triggered_time": { "order": "desc" } } ] } ``` You need to put in your watch ID though. This way we can see what was executed and if any errors occcured. Still wondering how it stopped executing. Also, did the logfiles on any of your nodes reveal anything about failed watch executions? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 26, 2020, 8:13am UTC](https://discuss.elastic.co/t/watch-is-stuck-at-error-always-needs-a-manual-restart/249778/6 "2020-10-26T08:13:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.