# Verifying that Watcher is set up fails
**URL:** https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [July 20, 2015, 6:58am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878 "2015-07-20T06:58:06Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@fzghm](https://discuss.elastic.co/u/fzghm)
#### Post date: [July 20, 2015, 6:58am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/1 "2015-07-20T06:58:06Z")
Getting a 400 error when try to verify watcher installation as follow:
% curl -XGET '[http://localhost:9200/\_watcher/\_stats?pretty](http://localhost:9200/_watcher/_stats?pretty)'
{
"error" : "ElasticsearchIllegalArgumentException[No feature for name [stats]]",
"status" : 404
}
Through trial & error I've changed the 'stats' in command to '\_stats' and get another error:
% curl -XGET '[http://localhost:9200/\_watcher/\_stats?pretty](http://localhost:9200/_watcher/_stats?pretty)'
{
"error" : "IndexMissingException[[\_watcher] missing]",
"status" : 404
}
What is the problem?
---
### Author:  [@PatrickKik](https://discuss.elastic.co/u/PatrickKik)
#### Post date: [July 20, 2015, 10:23am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/2 "2015-07-20T10:23:02Z")
If Elasticsearch tells you that the index \_watcher is missing, then the \_wachter endpoint is not registered.
Watcher is nog correctly installed. Have you restarted your node? In one of the first log messages it will tell you which plugins are loaded.
---
### Author:  [@fzghm](https://discuss.elastic.co/u/fzghm)
#### Post date: [July 26, 2015, 10:43am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/3 "2015-07-26T10:43:56Z")
I've restarted elasticsearch, but I don't know how to restart the nodes?
---
### Author:  [@PatrickKik](https://discuss.elastic.co/u/PatrickKik)
#### Post date: [July 27, 2015, 5:10am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/4 "2015-07-27T05:10:05Z")
If you restarted Elasticsearch, you are actually restarting a node. There should be a line like:
```
[2015-07-27 06:54:58,758][INFO][plugins] [Spoilsport] loaded [watcher], sites []
```
This will tell you the loaded plugins. How is yours looking?
---
### Author:  [@rvalta](https://discuss.elastic.co/u/rvalta)
#### Post date: [August 24, 2015, 9:44am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/5 "2015-08-24T09:44:02Z")
Is there any solution to this. Getting exactly the same problems when trying to verify whether watcher has been correctly installed. When I restart my Elasticsearch, I get the loaded plugins, but it looks like this:
---
### Author:  [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [August 24, 2015, 10:11am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/6 "2015-08-24T10:11:40Z")
If you don't see during startup that watcher has been loaded then it means watcher hasn't been installed. The run watcher successfully you need to install the watcher and license plugins on all nodes in your cluster like is done in the Watcher getting started.
To better verify if your required plugins are installed you can configure watcher to be a mandatory plugin in the elasticsearch.yml file. Like is configured here: [https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-plugins.html#\_mandatory\_plugins](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-plugins.html#_mandatory_plugins)
---
### Author:  [@Baboune](https://discuss.elastic.co/u/Baboune)
#### Post date: [August 26, 2015, 8:32am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/7 "2015-08-26T08:32:29Z")
Same issue: {"error":"ElasticsearchIllegalArgumentException[No feature for name [\_watcher]]","status":400}
Elasticsearch logs:
root@es-monitor-01:/usr/share/elasticsearch# tail -100f /var/log/elasticsearch/es-monitor-01/es-monitor-01.log
[2015-08-26 10:25:00,352][INFO][node] [es-monitor-01-es-monitor-01] version[1.5.2], pid[24338], build[62ff986/2015-04-27T09:21:06Z]
[2015-08-26 10:25:00,353][INFO][node] [es-monitor-01-es-monitor-01] initializing ...
[2015-08-26 10:25:00,414][INFO][plugins] [es-monitor-01-es-monitor-01] loaded [marvel, watcher, license], sites [marvel]
[2015-08-26 10:25:00,481][INFO][watcher.trigger.schedule] [es-monitor-01-es-monitor-01] using [ticker] schedule trigger engine
[2015-08-26 10:25:07,055][INFO][marvel.agent] [es-monitor-01-es-monitor-01] collecting disabled by settings
[2015-08-26 10:25:07,069][INFO][node] [es-monitor-01-es-monitor-01] initialized
[2015-08-26 10:25:07,069][INFO][node] [es-monitor-01-es-monitor-01] starting ...
[2015-08-26 10:25:07,149][INFO][transport] [es-monitor-01-es-monitor-01] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/20.20.21.13:9300]}
[2015-08-26 10:25:07,172][INFO][discovery] [es-monitor-01-es-monitor-01] es-monitor-01/hAnO5IaOQdGnR1sFSsrgpQ
[2015-08-26 10:25:10,973][INFO][cluster.service] [es-monitor-01-es-monitor-01] new\_master [es-monitor-01-es-monitor-01][hAnO5IaOQdGnR1sFSsrgpQ][es-monitor-01][inet[/20.20.21.13:9300]], reason: zen-disco-join (elected\_as\_master)
[2015-08-26 10:25:11,594][INFO][http] [es-monitor-01-es-monitor-01] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/20.20.21.13:9200]}
[2015-08-26 10:25:11,595][INFO][node] [es-monitor-01-es-monitor-01] started
[2015-08-26 10:25:13,683][INFO][license.plugin.core] [es-monitor-01-es-monitor-01] license for [watcher] - valid
[2015-08-26 10:25:13,693][ERROR][watcher.license] [es-monitor-01-es-monitor-01]
#
# Watcher license will expire on [Friday, September 25, 2015]. All configured actions on
# all registered watches are throttled (not executed) on Watcher license expiration.
# Watches will continue be evaluated and watch history will continue being recorded.
# Have a new license? please update it. Otherwise, please reach out to your support contact.
#
[2015-08-26 10:25:13,693][INFO][watcher] [es-monitor-01-es-monitor-01] starting watch service...
[2015-08-26 10:25:14,102][INFO][gateway] [es-monitor-01-es-monitor-01] recovered [15] indices into cluster\_state
[2015-08-26 10:25:14,215][INFO][watcher] [es-monitor-01-es-monitor-01] watch service has started
[2015-08-26 10:25:16,034][DEBUG][action.search.type] [es-monitor-01-es-monitor-01] All shards failed for phase: [query\_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [.marvel-2015.08.26][0] null
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:161)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:57)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:47)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
[...] at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Watcher seems to load but then there is this error. I tried to clean up all shards, restarted same issue.
Using elasticsearch-1.5.2.deb
---
### Author:  [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [August 26, 2015, 8:59am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/8 "2015-08-26T08:59:16Z")
When you log into the box where this node is running and run
curl 'localhost:9200/\_watcher'
then what is the response you see?
---
### Author:  [@Baboune](https://discuss.elastic.co/u/Baboune)
#### Post date: [August 26, 2015, 2:43pm UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/9 "2015-08-26T14:43:03Z")
Then I get:
curl 'localhost:9200/\_watcher'
{"version":{"name":"1.0.1","number":"1.0.1","build\_hash":"4dc4485f3e364c8f90da5476c150f8cda024e7ae","build\_timestamp":"2015-07-29T14:57:16Z","build\_snapshot":false}}
Which seems correct. Si what is the difference?
---
### Author:  [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [August 26, 2015, 2:54pm UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/10 "2015-08-26T14:54:40Z")
Did you connect to a node that didn't have Watcher installed?
(watcher needs to be installed in all nodes in your cluster)
---
### Author:  [@Baboune](https://discuss.elastic.co/u/Baboune)
#### Post date: [August 27, 2015, 6:54am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/11 "2015-08-27T06:54:48Z")
No, I did not connect to a node without watcher. This is a "monitoring" elastic search instance that is its own cluster (or a cluster of 1), This one monitoring node has watcher installed. There is a cluster of 5 nodes that are sending the monitoring data to that one node via marvel plugin.
It is deployed in a openstack cluster though so the external IP to reach the node is not the same as the internal IP.
What I then did: updated ES to 1.7.1, restarted, and now it appears to work. 😄
---
### Author:  [@gadin](https://discuss.elastic.co/u/gadin)
#### Post date: [September 29, 2015, 8:18am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/12 "2015-09-29T08:18:55Z")
I have the same issue with "{"error":"ElasticsearchIllegalArgumentException[No feature for name [\_watcher]]","status":400}"
The log shows
[2015-09-29 11:23:56,109][INFO][plugins] [Thog] loaded [marvel, license], sites [marvel]
The plugins list shows
~/elasticsearch-1.7.1 ❯❯❯ ./bin/plugin --list
Installed plugins:
- marvel
- license
- watcher
I didn't install any license though,
[http://172.23.14.36:9200/\_licenses:](http://172.23.14.36:9200/_licenses:)
{
"licenses" : []
}
No related error in the log
There isn't any jar file in the watcher folder though...
~/e/plugins ❯❯❯ find license -name \*.jar
license/elasticsearch-license-plugin-1.0.0.jar
~/e/plugins ❯❯❯ find watcher -name \*.jar
~/e/plugins ❯❯❯
---
### Author:  [@skearns](https://discuss.elastic.co/u/skearns)
#### Post date: [September 29, 2015, 8:41am UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/13 "2015-09-29T08:41:24Z")
Hi Gadin,
After installing the Watcher plugin on each node in your cluster, did you restart each node? Plugins are loaded at ES startup.
I hope that helps!
Steve
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:48pm UTC](https://discuss.elastic.co/t/verifying-that-watcher-is-set-up-fails/25878/14 "2017-07-06T13:48:26Z")