Verifying that Watcher is set up fails

Elastic Stack Elasticsearch
1

Getting a 400 error when try to verify watcher installation as follow:
% curl -XGET 'http://localhost:9200/_watcher/_stats?pretty'

{
"error" : "ElasticsearchIllegalArgumentException[No feature for name [stats]]",
"status" : 404
}

Through trial & error I've changed the 'stats' in command to '_stats' and get another error:
% curl -XGET 'http://localhost:9200/_watcher/_stats?pretty'

{
"error" : "IndexMissingException[[_watcher] missing]",
"status" : 404
}

What is the problem?

2

If Elasticsearch tells you that the index _watcher is missing, then the _wachter endpoint is not registered.
Watcher is nog correctly installed. Have you restarted your node? In one of the first log messages it will tell you which plugins are loaded.

3

I've restarted elasticsearch, but I don't know how to restart the nodes?

4

If you restarted Elasticsearch, you are actually restarting a node. There should be a line like:

[2015-07-27 06:54:58,758][INFO ][plugins                  ] [Spoilsport] loaded [watcher], sites []

This will tell you the loaded plugins. How is yours looking?

5

Is there any solution to this. Getting exactly the same problems when trying to verify whether watcher has been correctly installed. When I restart my Elasticsearch, I get the loaded plugins, but it looks like this:

6

If you don't see during startup that watcher has been loaded then it means watcher hasn't been installed. The run watcher successfully you need to install the watcher and license plugins on all nodes in your cluster like is done in the Watcher getting started.

To better verify if your required plugins are installed you can configure watcher to be a mandatory plugin in the elasticsearch.yml file. Like is configured here: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-plugins.html#_mandatory_plugins

7

Same issue: {"error":"ElasticsearchIllegalArgumentException[No feature for name [_watcher]]","status":400}

Elasticsearch logs:
root@es-monitor-01:/usr/share/elasticsearch# tail -100f /var/log/elasticsearch/es-monitor-01/es-monitor-01.log
[2015-08-26 10:25:00,352][INFO ][node ] [es-monitor-01-es-monitor-01] version[1.5.2], pid[24338], build[62ff986/2015-04-27T09:21:06Z]
[2015-08-26 10:25:00,353][INFO ][node ] [es-monitor-01-es-monitor-01] initializing ...
[2015-08-26 10:25:00,414][INFO ][plugins ] [es-monitor-01-es-monitor-01] loaded [marvel, watcher, license], sites [marvel]
[2015-08-26 10:25:00,481][INFO ][watcher.trigger.schedule ] [es-monitor-01-es-monitor-01] using [ticker] schedule trigger engine
[2015-08-26 10:25:07,055][INFO ][marvel.agent ] [es-monitor-01-es-monitor-01] collecting disabled by settings
[2015-08-26 10:25:07,069][INFO ][node ] [es-monitor-01-es-monitor-01] initialized
[2015-08-26 10:25:07,069][INFO ][node ] [es-monitor-01-es-monitor-01] starting ...
[2015-08-26 10:25:07,149][INFO ][transport ] [es-monitor-01-es-monitor-01] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/20.20.21.13:9300]}
[2015-08-26 10:25:07,172][INFO ][discovery ] [es-monitor-01-es-monitor-01] es-monitor-01/hAnO5IaOQdGnR1sFSsrgpQ
[2015-08-26 10:25:10,973][INFO ][cluster.service ] [es-monitor-01-es-monitor-01] new_master [es-monitor-01-es-monitor-01][hAnO5IaOQdGnR1sFSsrgpQ][es-monitor-01][inet[/20.20.21.13:9300]], reason: zen-disco-join (elected_as_master)
[2015-08-26 10:25:11,594][INFO ][http ] [es-monitor-01-es-monitor-01] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/20.20.21.13:9200]}
[2015-08-26 10:25:11,595][INFO ][node ] [es-monitor-01-es-monitor-01] started
[2015-08-26 10:25:13,683][INFO ][license.plugin.core ] [es-monitor-01-es-monitor-01] license for [watcher] - valid
[2015-08-26 10:25:13,693][ERROR][watcher.license ] [es-monitor-01-es-monitor-01]

Watcher license will expire on [Friday, September 25, 2015]. All configured actions on

all registered watches are throttled (not executed) on Watcher license expiration.

Watches will continue be evaluated and watch history will continue being recorded.

Have a new license? please update it. Otherwise, please reach out to your support contact.

[2015-08-26 10:25:13,693][INFO ][watcher ] [es-monitor-01-es-monitor-01] starting watch service...
[2015-08-26 10:25:14,102][INFO ][gateway ] [es-monitor-01-es-monitor-01] recovered [15] indices into cluster_state
[2015-08-26 10:25:14,215][INFO ][watcher ] [es-monitor-01-es-monitor-01] watch service has started
[2015-08-26 10:25:16,034][DEBUG][action.search.type ] [es-monitor-01-es-monitor-01] All shards failed for phase: [query_fetch]
org.elasticsearch.action.NoShardAvailableActionException: [.marvel-2015.08.26][0] null
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:161)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:57)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction.doExecute(TransportSearchQueryAndFetchAction.java:47)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)

[...] at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Watcher seems to load but then there is this error. I tried to clean up all shards, restarted same issue.

Using elasticsearch-1.5.2.deb

8

When you log into the box where this node is running and run
curl 'localhost:9200/_watcher'

then what is the response you see?

9

Then I get:

curl 'localhost:9200/_watcher'
{"version":{"name":"1.0.1","number":"1.0.1","build_hash":"4dc4485f3e364c8f90da5476c150f8cda024e7ae","build_timestamp":"2015-07-29T14:57:16Z","build_snapshot":false}}

Which seems correct. Si what is the difference?

10

Did you connect to a node that didn't have Watcher installed?
(watcher needs to be installed in all nodes in your cluster)

11

No, I did not connect to a node without watcher. This is a "monitoring" elastic search instance that is its own cluster (or a cluster of 1), This one monitoring node has watcher installed. There is a cluster of 5 nodes that are sending the monitoring data to that one node via marvel plugin.

It is deployed in a openstack cluster though so the external IP to reach the node is not the same as the internal IP.

What I then did: updated ES to 1.7.1, restarted, and now it appears to work. :smile:

12

I have the same issue with "{"error":"ElasticsearchIllegalArgumentException[No feature for name [_watcher]]","status":400}"

The log shows
[2015-09-29 11:23:56,109][INFO ][plugins ] [Thog] loaded [marvel, license], sites [marvel]

The plugins list shows
~/elasticsearch-1.7.1 ❯❯❯ ./bin/plugin --list
Installed plugins:
- marvel
- license
- watcher

I didn't install any license though,
http://172.23.14.36:9200/_licenses:
{
"licenses" : [ ]
}

No related error in the log

There isn't any jar file in the watcher folder though...
~/e/plugins ❯❯❯ find license -name *.jar
license/elasticsearch-license-plugin-1.0.0.jar
~/e/plugins ❯❯❯ find watcher -name *.jar
~/e/plugins ❯❯❯

13

Hi Gadin,

After installing the Watcher plugin on each node in your cluster, did you restart each node? Plugins are loaded at ES startup.

I hope that helps!
Steve