# How to create \_watcher index

**URL:** <https://discuss.elastic.co/t/how-to-create--watcher-index/72523>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [January 23, 2017, 6:21pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523 "2017-01-23T18:21:06Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [January 23, 2017, 6:21pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/1 "2017-01-23T18:21:06Z")

</div>

Hi Team,  
Please suggest how can i create/start watcher index? i am getting below error:

i deleted watcher index to resolve an issue and its not working since then, i updated elasticsearch.yml for: watcher.index.rest.direct\_access: true

{  
"error": {  
"root\_cause": [  
{  
"type": "remote\_transport\_exception",  
"reason": "[es-master-node][1.1.1.1:9300][cluster:admin/watcher/watch/put]"  
}  
],  
"type": "illegal\_state\_exception",  
"reason": "not started"  
},  
"status": 500  
}

GET .watches/\_stats: (doesn't show any failure)

GET \_watcher/\_stats  
{  
"error": {  
"root\_cause": [  
{  
"type": "index\_not\_found\_exception",  
"reason": "no such index",  
"index": "\_watcher",  
"resource.type": "index\_or\_alias",  
"[resource.id](http://resource.id)": "\_watcher"  
}  
],  
"type": "index\_not\_found\_exception",  
"reason": "no such index",  
"index": "\_watcher",  
"resource.type": "index\_or\_alias",  
"[resource.id](http://resource.id)": "\_watcher"  
},  
"status": 404  
}

Thanks & Regards

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 24, 2017, 8:13am UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/2 "2017-01-24T08:13:27Z")

</div>

Hey,

have you tried the [Start API](https://www.elastic.co/guide/en/watcher/2.4/api-rest.html#api-rest-start)?

--Alex

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [January 24, 2017, 4:59pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/3 "2017-01-24T16:59:13Z")

</div>

No, but this is what i got:

[2017-01-24 16:00:27,956][WARN][rest.suppressed] /\_watcher/\_start Params: {}  
RemoteTransportException[[es-master-node][1.1.1.1:9300][cluster:admin/watcher/service]]; nested: NullPointerException;  
Caused by: java.lang.NullPointerException  
at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87)  
at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109)  
at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97)  
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97)  
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64)  
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:67)  
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38)  
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90)  
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at java.lang.Thread.run(Thread.java:745)

Tried:

PUT \_watcher/\_stop  
PUT \_watcher/\_start  
GET \_watcher/stats:  
{  
"watcher\_state": "stopped",  
"watch\_count": 0,  
"execution\_thread\_pool": {  
"queue\_size": 0,  
"max\_size": 0  
},  
"manually\_stopped": true  
}

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 25, 2017, 5:13pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/4 "2017-01-25T17:13:45Z")

</div>

Hey,

which elasticsearch version is this?

--Alex

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [January 25, 2017, 5:26pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/5 "2017-01-25T17:26:17Z")

</div>

ES Version: 2.3.1

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 25, 2017, 8:59pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/6 "2017-01-25T20:59:47Z")

</div>

Hey,

do you have any unassigned shards of the `.triggered_watches` index? You can try to delete that index and then restart watcher.

--Alex

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [January 26, 2017, 11:56pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/7 "2017-01-26T23:56:13Z")

</div>

Delete .triggered\_watches-\* and .watches\_history-\* , also deleted their index pattern,

But when try to start, got same error:

PUT \_watcher/\_start:

{  
"error": {  
"root\_cause": [  
{  
"type": "remote\_transport\_exception",  
"reason": "[es-master-node][1.1.1.1:9300][cluster:admin/watcher/service]"  
}  
],  
"type": "null\_pointer\_exception",  
"reason": null  
},  
"status": 500  
}

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 27, 2017, 8:11am UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/8 "2017-01-27T08:11:28Z")

</div>

Can you paste the output of the [cat shards API](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/cat-shards.html) please?

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [January 27, 2017, 10:31pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/9 "2017-01-27T22:31:53Z")

</div>

That would be a long list, attached might help:

 ![](https://us1.discourse-cdn.com/elastic/original/2X/a/a4f6f3e29d1d485a5eadf46da88a339a54d59c6f.png)

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 28, 2017, 11:44pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/10 "2017-01-28T23:44:33Z")

</div>

Hey,

yes, that was actually helpful. No shards are unassigned, thats what I wanted to know.

Can you try the following

- Ensure that the index template `triggered_watches` exists
- Create an index named `.triggered_watches`
- Ensure that the index is in an green state
- Restart watcher again
- Paste the exception message from the master log if occured

--Alex

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 30, 2017, 8:24am UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/11 "2017-01-30T08:24:12Z")

</div>

One more thing, can you share your cluster state? It's going to be big, so it would be nice, if you can put it into some gist or something.

In addition, can you paste the output from

```auto
GET _cat/indices/.triggered-watches

```

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 30, 2017, 10:14am UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/12 "2017-01-30T10:14:22Z")

</div>

Is is possible, that your `.triggered_watches` index is closed?

Can you open it again?

Also note, that this needs to be `.triggered_watches` with an underscore instead of a dash

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [January 30, 2017, 10:56pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/13 "2017-01-30T22:56:59Z")

</div>

Yup, resolved. The template was missing.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2017, 10:57pm UTC](https://discuss.elastic.co/t/how-to-create--watcher-index/72523/14 "2017-02-27T22:57:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.