# On watcher installation, watcher\_state status is stopped

**URL:** https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [July 13, 2015, 8:30am UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419 "2015-07-13T08:30:46Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Shubham\_Bhartiya](https://avatars.discourse-cdn.com/v4/letter/s/9dc877/32.png) [@Shubham\_Bhartiya](https://discuss.elastic.co/u/Shubham_Bhartiya)
#### Post date: [July 13, 2015, 8:30am UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/1 "2015-07-13T08:30:46Z")

</div>

I have a two node cluster having ElasticSearch version 1.5.2 on which I have installed watcher plugin. After installation, there is nothing mentioned in the logs but still the cluster health appeared red.  
Also, if I use this command curl -XGET '[http://localhost:9200/\_watcher/stats?pretty](http://localhost:9200/_watcher/stats?pretty)' and get the watcher stats, The output is  
{  
"watcher\_state" : "stopped",  
"watch\_count" : 0,  
"execution\_thread\_pool" : {  
"queue\_size" : 0,  
"max\_size" : 0  
}  
}  
The watcher\_state is stopped rather than starting.

---

<div class="post-metadata">

### Author: ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [July 13, 2015, 9:01am UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/2 "2015-07-13T09:01:52Z")

</div>

If the cluster state is red, then it is likely that Watcher can't start too.

If you check the shards cat api (host:9200/\_cat/shards), do you see shards of the .watches, .triggered\_watches or .watch\_history-\* indices unassigned?

---

<div class="post-metadata">

### Author: ![Shubham\_Bhartiya](https://avatars.discourse-cdn.com/v4/letter/s/9dc877/32.png) [@Shubham\_Bhartiya](https://discuss.elastic.co/u/Shubham_Bhartiya)
#### Post date: [July 13, 2015, 11:17am UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/3 "2015-07-13T11:17:58Z")

</div>

![](https://us1.discourse-cdn.com/elastic/original/2X/2/280528a68d131915efab86e776d7826f2d0602cb.png) ![](https://us1.discourse-cdn.com/elastic/original/2X/8/84e7486ff35339a88ba82bf3db9eadb306a4d088.png)

The first image displays the cluster health and watcher stats.  
Second image is the host:9200/\_cat/shards. I cannot see the indices unassigned...

---

<div class="post-metadata">

### Author: ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [July 13, 2015, 12:33pm UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/4 "2015-07-13T12:33:06Z")

</div>

Can you grep the shard cat output?  
(curl 'localhost:9200/\_cat/shards' | grep watch'

Also can you set the watcher logging to debug and restart your nodes and share what you see in the log files?  
(add: `watcher: DEBUG` under the `logger` part in the logging.yml file)

---

<div class="post-metadata">

### Author: ![Shubham\_Bhartiya](https://avatars.discourse-cdn.com/v4/letter/s/9dc877/32.png) [@Shubham\_Bhartiya](https://discuss.elastic.co/u/Shubham_Bhartiya)
#### Post date: [July 13, 2015, 1:19pm UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/5 "2015-07-13T13:19:13Z")

</div>

In logs, I am getting the following output.  
[2015-07-13 13:08:05,002][DEBUG][watcher] [Franz Kafka] not starting watcher. because the cluster isn't ready yet to run watcher

And the grep watch image...

 ![](https://us1.discourse-cdn.com/elastic/original/2X/0/0c8383571c50031f9a2d18b0c1f276bed1a77bad.png)

---

<div class="post-metadata">

### Author: ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [July 13, 2015, 1:37pm UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/6 "2015-07-13T13:37:40Z")

</div>

So all the shards of `.watches` index are unassigned and preventing watcher from starting.  
I don't why these shards are unassigned, because there is nothing in logs about this.

If you have your watches available elsewhere you can just remove the `.watches` index, wait for watcher to start and then re-instert your watches. Watcher should then be operational.

---

<div class="post-metadata">

### Author: ![Shubham\_Bhartiya](https://avatars.discourse-cdn.com/v4/letter/s/9dc877/32.png) [@Shubham\_Bhartiya](https://discuss.elastic.co/u/Shubham_Bhartiya)
#### Post date: [July 13, 2015, 2:05pm UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/7 "2015-07-13T14:05:43Z")

</div>

Yes, deleting .watches index turned the cluster status to green. Now the things are working perfectly.  
Thanks for your help.  
Btw, I used this command to delete the .watches index curl -XDELETE '[http://localhost:9200/\*watches](http://localhost:9200/*watches)' -v

---

<div class="post-metadata">

### Author: ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [July 13, 2015, 3:42pm UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/8 "2015-07-13T15:42:37Z")

</div>

that curl command invokes the delete index api, so that is good.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:49pm UTC](https://discuss.elastic.co/t/on-watcher-installation-watcher-state-status-is-stopped/25419/9 "2017-07-06T13:49:18Z")

</div>


