# Trying out Watcher but once installed, Elasticsearch feed stops
**URL:** https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [July 10, 2015, 4:10pm UTC](https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332 "2015-07-10T16:10:01Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@dmccuk](https://discuss.elastic.co/u/dmccuk)
#### Post date: [July 10, 2015, 4:10pm UTC](https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332/1 "2015-07-10T16:10:01Z")
Hi,
I'm testing out watcher to see if it's viable to use as an alternative to out current monitoring systems. The issue I'm having is once I install watched (+license) and restart elasticsearch, my feed stops and I can't get it working again.
If i uninstall watched +License and restart ES + Logstash it works again. I'll post my log files on Monday morning but after checking the watcher download page, the only requisite i can see is ES v1.6. The versions I'm running for the Stack are:
ES 1.6
LS 1.4.2
Redis 2.4
Kibana 3
Does anyone have any advice or know If I'm missing something obvious. I'll load up some logs on Monday. Need to leave...
Thanks - Den
---
### Author:  [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [July 10, 2015, 5:17pm UTC](https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332/2 "2015-07-10T17:17:34Z")
Would be great if you can share the watch itself and the log stash config too for some more context.
---
### Author:  [@dmccuk](https://discuss.elastic.co/u/dmccuk)
#### Post date: [July 14, 2015, 11:06am UTC](https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332/3 "2015-07-14T11:06:21Z")
Sorry about the delay. Yes so here' the setup:
Doing an offline install of the License and watcher:
# bin/plugin -u file:///tmp/license-latest.zip -i elasticsearch/license/latest
-\> Installing elasticsearch/license/latest...
Trying file:/tmp/license-latest.zip...
Downloading .DONE
Installed elasticsearch/license/latest into /usr/share/elasticsearch/plugins/license
# bin/plugin -u file:///tmp/watcher-latest.zip -i elasticsearch/watcher/latest
-\> Installing elasticsearch/watcher/latest...
Trying file:/tmp/watcher-latest.zip...
Downloading .......................................DONE
Installed elasticsearch/watcher/latest into /usr/share/elasticsearch/plugins/watcher
So my Index size is increasing:
# du -sk /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14
58308 /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14
# du -sk /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14
60024 /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14
Restarting after the install:
# date
Tue Jul 14 11:49:32 BST 2015
# service elasticsearch restart
Stopping elasticsearch: [OK]
Starting elasticsearch: [OK]
#
Checking Watcher:
# tail -f /var/log/elasticsearch/elasticsearch.log
[2015-07-14 11:50:08,541][INFO][gateway] [Gronk] recovered [3] indices into cluster\_state
[2015-07-14 11:50:09,053][INFO][license.plugin.core] [Gronk] license for [watcher] - valid
[2015-07-14 11:50:09,066][INFO][watcher] [Gronk] watch service has started
[2015-07-14 11:50:09,070][ERROR][watcher.license] [Gronk]
#
# Watcher license will expire on [Thursday, August 13, 2015]. All configured actions on
# all registered watches are throttled (not executed) on Watcher license expiration.
# Watches will continue be evaluated and watch history will continue being recorded.
# Have a new license? please update it. Otherwise, please reach out to your support contact.
#
# curl -XGET '[http://localhost:9200/\_watcher/stats?pretty](http://localhost:9200/_watcher/stats?pretty)'
{
"watcher\_state" : "started",
"watch\_count" : 0,
"execution\_thread\_pool" : {
"queue\_size" : 0,
"max\_size" : 0
}
}
#
so the feed has stopped. My logstash config is quite large but the output section only has this in it:
##############################################
# OUTPUT SECTION
# This is where logstash sends the data over to ElasticSearch
##############################################
output {
elasticsearch { host =\> localhost }
I also have the output going to stdout:
##############################################
# STDOUT is used for debug purposes to check your messages are coming in.
# uncomment and tail /var/log/logstash/logstash.stdout
##############################################
stdout {
codec =\> json #rubydebug
}
After another restart of elasticsearch i get this:
[2015-07-14 12:01:56,250][INFO][node] [Gronk] stopping ...
[2015-07-14 12:01:56,255][INFO][watcher] [Gronk] stopping watch service...
[2015-07-14 12:01:56,648][INFO][watcher.watch] [Gronk] stopped watch store
[2015-07-14 12:01:56,648][INFO][watcher] [Gronk] watch service has stopped
[2015-07-14 12:01:56,699][INFO][node] [Gronk] stopped
[2015-07-14 12:01:56,700][INFO][node] [Gronk] closing ...
[2015-07-14 12:01:56,704][INFO][node] [Gronk] closed
[2015-07-14 12:01:57,822][INFO][node] [Jaren] version[1.6.0], pid[9736], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-07-14 12:01:57,823][INFO][node] [Jaren] initializing ...
[2015-07-14 12:01:57,858][INFO][plugins] [Jaren] loaded [license, watcher], sites []
[2015-07-14 12:01:57,900][INFO][env] [Jaren] using [1] data paths, mounts [[/var (/dev/mapper/VGv0-LV\_var)]], net usable\_space [13.7gb], net total\_space [15.7gb], types [ext4]
[2015-07-14 12:01:57,934][INFO][watcher.trigger.schedule] [Jaren] using [ticker] schedule trigger engine
[2015-07-14 12:02:00,415][INFO][node] [Jaren] initialized
[2015-07-14 12:02:00,415][INFO][node] [Jaren] starting ...
[2015-07-14 12:02:00,470][INFO][transport] [Jaren] bound\_address {inet[/0.0.0.0:9300]}, publish\_address {inet[/111.111.111.111:9300]}
[2015-07-14 12:02:00,480][INFO][discovery] [Jaren] elasticsearch/bZneklHbTua7zPeRRD69uQ
[2015-07-14 12:02:04,247][INFO][cluster.service] [Jaren] new\_master [Jaren][bZneklHbTua7zPeRRD69uQ][server1][inet[/111.111.111.111:9300]], reason: zen-disco-join (elected\_as\_master)
[2015-07-14 12:02:04,300][INFO][http] [Jaren] bound\_address {inet[/0.0.0.0:9200]}, publish\_address {inet[/111.111.111.111:9200]}
[2015-07-14 12:02:04,300][INFO][node] [Jaren] started
[2015-07-14 12:02:04,319][INFO][watcher] [Jaren] starting watch service...
[2015-07-14 12:02:04,323][INFO][license.plugin.core] [Jaren] license for [watcher] - valid
[2015-07-14 12:02:04,337][ERROR][watcher.license] [Jaren]
#
# Watcher license will expire on [Thursday, August 13, 2015]. All configured actions on
# all registered watches are throttled (not executed) on Watcher license expiration.
# Watches will continue be evaluated and watch history will continue being recorded.
# Have a new license? please update it. Otherwise, please reach out to your support contact.
#
[2015-07-14 12:02:04,362][INFO][gateway] [Jaren] recovered [3] indices into cluster\_state
[2015-07-14 12:02:04,952][INFO][watcher] [Jaren] watch service has started
Does this help?
If i remove watcher and the license and restart, the feed comes back again.
---
### Author:  [@dmccuk](https://discuss.elastic.co/u/dmccuk)
#### Post date: [July 14, 2015, 11:12am UTC](https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332/4 "2015-07-14T11:12:43Z")
oh, by the way this is a single node cluster. I've tried restarting logstash. I get this message repeating in the logstash.log:
log4j, [2015-07-14T12:11:20.218] WARN: org.elasticsearch.transport.netty: [server1-9243-4232] Message not fully read (request) for [191] and action [discovery/zen/join/validate], resetting
log4j, [2015-07-14T12:11:23.227] WARN: org.elasticsearch.transport.netty: [server1-9243-4232] Message not fully read (request) for [192] and action [discovery/zen/join/validate], resetting
Regards
---
### Author:  [@dmccuk](https://discuss.elastic.co/u/dmccuk)
#### Post date: [July 14, 2015, 11:26am UTC](https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332/5 "2015-07-14T11:26:01Z")
Actually I think I fixed it. Checking that alert pushed me to find this post:
> [@After install watcher logstash cannot join the cluster](https://discuss.elastic.co/t/after-install-watcher-logstash-cannot-join-the-cluster/1744):
>
> Hello good people, I have 3 nodes cluster and 2 logstash instance use elasticsearch node output. After installing watcher for all 3 nodes, the logstash instances repeatedly showing the following warning and cannot do anything more. WARN: org.elasticsearch.transport.netty: [logstash-\*\*\*\*\*\*\*5-6181-2044] Message not fully read (request) for [4860] and action [discovery/zen/join/validate], resetting INFO: org.elasticsearch.discovery.zen: [logstash-\*\*\*\*\*\*\*5-6181-2044] failed to send join request t…
I updated my output protocol and it kicked back into life. Kibana3 still wasn't showing me the output so i change this line back to port 9200 and the data appeared.
```
elasticsearch: "http://"+window.location.hostname+":9200",
```
Hopefully i can get on with testing watcher now.
Regards
Dennis
---
### Author:  [@mvg](https://discuss.elastic.co/u/mvg)
#### Post date: [July 14, 2015, 2:39pm UTC](https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332/6 "2015-07-14T14:39:23Z")
Correct, elasticsearch output should be used. Happy testing 😄
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:49pm UTC](https://discuss.elastic.co/t/trying-out-watcher-but-once-installed-elasticsearch-feed-stops/25332/7 "2017-07-06T13:49:15Z")