Trying out Watcher but once installed, Elasticsearch feed stops

Hi,

I'm testing out watcher to see if it's viable to use as an alternative to out current monitoring systems. The issue I'm having is once I install watched (+license) and restart elasticsearch, my feed stops and I can't get it working again.

If i uninstall watched +License and restart ES + Logstash it works again. I'll post my log files on Monday morning but after checking the watcher download page, the only requisite i can see is ES v1.6. The versions I'm running for the Stack are:

ES 1.6
LS 1.4.2
Redis 2.4
Kibana 3

Does anyone have any advice or know If I'm missing something obvious. I'll load up some logs on Monday. Need to leave...

Thanks - Den

Would be great if you can share the watch itself and the log stash config too for some more context.

Sorry about the delay. Yes so here' the setup:

Doing an offline install of the License and watcher:

bin/plugin -u file:///tmp/license-latest.zip -i elasticsearch/license/latest

-> Installing elasticsearch/license/latest...
Trying file:/tmp/license-latest.zip...
Downloading .DONE
Installed elasticsearch/license/latest into /usr/share/elasticsearch/plugins/license

bin/plugin -u file:///tmp/watcher-latest.zip -i elasticsearch/watcher/latest

-> Installing elasticsearch/watcher/latest...
Trying file:/tmp/watcher-latest.zip...
Downloading .......................................DONE
Installed elasticsearch/watcher/latest into /usr/share/elasticsearch/plugins/watcher

So my Index size is increasing:

du -sk /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14

58308 /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14

du -sk /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14

60024 /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14

Restarting after the install:

date

Tue Jul 14 11:49:32 BST 2015

service elasticsearch restart

Stopping elasticsearch: [ OK ]
Starting elasticsearch: [ OK ]

Checking Watcher:

tail -f /var/log/elasticsearch/elasticsearch.log

[2015-07-14 11:50:08,541][INFO ][gateway ] [Gronk] recovered [3] indices into cluster_state
[2015-07-14 11:50:09,053][INFO ][license.plugin.core ] [Gronk] license for [watcher] - valid
[2015-07-14 11:50:09,066][INFO ][watcher ] [Gronk] watch service has started
[2015-07-14 11:50:09,070][ERROR][watcher.license ] [Gronk]

Watcher license will expire on [Thursday, August 13, 2015]. All configured actions on

all registered watches are throttled (not executed) on Watcher license expiration.

Watches will continue be evaluated and watch history will continue being recorded.

Have a new license? please update it. Otherwise, please reach out to your support contact.

curl -XGET 'http://localhost:9200/_watcher/stats?pretty'

{
"watcher_state" : "started",
"watch_count" : 0,
"execution_thread_pool" : {
"queue_size" : 0,
"max_size" : 0
}
}

so the feed has stopped. My logstash config is quite large but the output section only has this in it:

##############################################

OUTPUT SECTION

This is where logstash sends the data over to ElasticSearch

##############################################

output {
elasticsearch { host => localhost }

I also have the output going to stdout:

##############################################

STDOUT is used for debug purposes to check your messages are coming in.

uncomment and tail /var/log/logstash/logstash.stdout

##############################################
stdout {
codec => json #rubydebug
}

After another restart of elasticsearch i get this:

[2015-07-14 12:01:56,250][INFO ][node ] [Gronk] stopping ...
[2015-07-14 12:01:56,255][INFO ][watcher ] [Gronk] stopping watch service...
[2015-07-14 12:01:56,648][INFO ][watcher.watch ] [Gronk] stopped watch store
[2015-07-14 12:01:56,648][INFO ][watcher ] [Gronk] watch service has stopped
[2015-07-14 12:01:56,699][INFO ][node ] [Gronk] stopped
[2015-07-14 12:01:56,700][INFO ][node ] [Gronk] closing ...
[2015-07-14 12:01:56,704][INFO ][node ] [Gronk] closed
[2015-07-14 12:01:57,822][INFO ][node ] [Jaren] version[1.6.0], pid[9736], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-07-14 12:01:57,823][INFO ][node ] [Jaren] initializing ...
[2015-07-14 12:01:57,858][INFO ][plugins ] [Jaren] loaded [license, watcher], sites []
[2015-07-14 12:01:57,900][INFO ][env ] [Jaren] using [1] data paths, mounts [[/var (/dev/mapper/VGv0-LV_var)]], net usable_space [13.7gb], net total_space [15.7gb], types [ext4]
[2015-07-14 12:01:57,934][INFO ][watcher.trigger.schedule ] [Jaren] using [ticker] schedule trigger engine
[2015-07-14 12:02:00,415][INFO ][node ] [Jaren] initialized
[2015-07-14 12:02:00,415][INFO ][node ] [Jaren] starting ...
[2015-07-14 12:02:00,470][INFO ][transport ] [Jaren] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/111.111.111.111:9300]}
[2015-07-14 12:02:00,480][INFO ][discovery ] [Jaren] elasticsearch/bZneklHbTua7zPeRRD69uQ
[2015-07-14 12:02:04,247][INFO ][cluster.service ] [Jaren] new_master [Jaren][bZneklHbTua7zPeRRD69uQ][server1][inet[/111.111.111.111:9300]], reason: zen-disco-join (elected_as_master)
[2015-07-14 12:02:04,300][INFO ][http ] [Jaren] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/111.111.111.111:9200]}
[2015-07-14 12:02:04,300][INFO ][node ] [Jaren] started
[2015-07-14 12:02:04,319][INFO ][watcher ] [Jaren] starting watch service...
[2015-07-14 12:02:04,323][INFO ][license.plugin.core ] [Jaren] license for [watcher] - valid
[2015-07-14 12:02:04,337][ERROR][watcher.license ] [Jaren]

Watcher license will expire on [Thursday, August 13, 2015]. All configured actions on

all registered watches are throttled (not executed) on Watcher license expiration.

Watches will continue be evaluated and watch history will continue being recorded.

Have a new license? please update it. Otherwise, please reach out to your support contact.

[2015-07-14 12:02:04,362][INFO ][gateway ] [Jaren] recovered [3] indices into cluster_state
[2015-07-14 12:02:04,952][INFO ][watcher ] [Jaren] watch service has started

Does this help?

If i remove watcher and the license and restart, the feed comes back again.

oh, by the way this is a single node cluster. I've tried restarting logstash. I get this message repeating in the logstash.log:

log4j, [2015-07-14T12:11:20.218] WARN: org.elasticsearch.transport.netty: [server1-9243-4232] Message not fully read (request) for [191] and action [discovery/zen/join/validate], resetting
log4j, [2015-07-14T12:11:23.227] WARN: org.elasticsearch.transport.netty: [server1-9243-4232] Message not fully read (request) for [192] and action [discovery/zen/join/validate], resetting

Regards

Actually I think I fixed it. Checking that alert pushed me to find this post:

I updated my output protocol and it kicked back into life. Kibana3 still wasn't showing me the output so i change this line back to port 9200 and the data appeared.

elasticsearch: "http://"+window.location.hostname+":9200",

Hopefully i can get on with testing watcher now.

Regards

Dennis

Correct, elasticsearch output should be used. Happy testing :smile: