Trying out Watcher but once installed, Elasticsearch feed stops

[dmccuk (Dennis)]

Hi,

I'm testing out watcher to see if it's viable to use as an alternative to out current monitoring systems. The issue I'm having is once I install watched (+license) and restart elasticsearch, my feed stops and I can't get it working again.

If i uninstall watched +License and restart ES + Logstash it works again. I'll post my log files on Monday morning but after checking the watcher download page, the only requisite i can see is ES v1.6. The versions I'm running for the Stack are:

ES 1.6
LS 1.4.2
Redis 2.4
Kibana 3

Does anyone have any advice or know If I'm missing something obvious. I'll load up some logs on Monday. Need to leave...

Thanks - Den

[mvg (Martijn Van Groningen)]

Would be great if you can share the watch itself and the log stash config too for some more context.

[dmccuk (Dennis)]

Sorry about the delay. Yes so here' the setup:

Doing an offline install of the License and watcher:

bin/plugin -u file:///tmp/license-latest.zip -i elasticsearch/license/latest

-> Installing elasticsearch/license/latest...
Trying file:/tmp/license-latest.zip...
Downloading .DONE
Installed elasticsearch/license/latest into /usr/share/elasticsearch/plugins/license

bin/plugin -u file:///tmp/watcher-latest.zip -i elasticsearch/watcher/latest

-> Installing elasticsearch/watcher/latest...
Trying file:/tmp/watcher-latest.zip...
Downloading .......................................DONE
Installed elasticsearch/watcher/latest into /usr/share/elasticsearch/plugins/watcher

So my Index size is increasing:

du -sk /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14

58308 /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14

du -sk /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14

60024 /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.07.14

Restarting after the install:

date

Tue Jul 14 11:49:32 BST 2015

service elasticsearch restart

Stopping elasticsearch: [ OK ]
Starting elasticsearch: [ OK ]

Checking Watcher:

tail -f /var/log/elasticsearch/elasticsearch.log

[2015-07-14 11:50:08,541][INFO ][gateway ] [Gronk] recovered [3] indices into cluster_state
[2015-07-14 11:50:09,053][INFO ][license.plugin.core ] [Gronk] license for [watcher] - valid
[2015-07-14 11:50:09,066][INFO ][watcher ] [Gronk] watch service has started
[2015-07-14 11:50:09,070][ERROR][watcher.license ] [Gronk]

Watcher license will expire on [Thursday, August 13, 2015]. All configured actions on

all registered watches are throttled (not executed) on Watcher license expiration.

Watches will continue be evaluated and watch history will continue being recorded.

Have a new license? please update it. Otherwise, please reach out to your support contact.

curl -XGET 'http://localhost:9200/_watcher/stats?pretty'

{
"watcher_state" : "started",
"watch_count" : 0,
"execution_thread_pool" : {
"queue_size" : 0,
"max_size" : 0
}
}

so the feed has stopped. My logstash config is quite large but the output section only has this in it:

##############################################

OUTPUT SECTION

This is where logstash sends the data over to ElasticSearch

##############################################

output {
elasticsearch { host => localhost }

I also have the output going to stdout:

##############################################

STDOUT is used for debug purposes to check your messages are coming in.

uncomment and tail /var/log/logstash/logstash.stdout

##############################################
stdout {
codec => json #rubydebug
}

After another restart of elasticsearch i get this:

[2015-07-14 12:01:56,250][INFO ][node ] [Gronk] stopping ...
[2015-07-14 12:01:56,255][INFO ][watcher ] [Gronk] stopping watch service...
[2015-07-14 12:01:56,648][INFO ][watcher.watch ] [Gronk] stopped watch store
[2015-07-14 12:01:56,648][INFO ][watcher ] [Gronk] watch service has stopped
[2015-07-14 12:01:56,699][INFO ][node ] [Gronk] stopped
[2015-07-14 12:01:56,700][INFO ][node ] [Gronk] closing ...
[2015-07-14 12:01:56,704][INFO ][node ] [Gronk] closed
[2015-07-14 12:01:57,822][INFO ][node ] [Jaren] version[1.6.0], pid[9736], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-07-14 12:01:57,823][INFO ][node ] [Jaren] initializing ...
[2015-07-14 12:01:57,858][INFO ][plugins ] [Jaren] loaded [license, watcher], sites []
[2015-07-14 12:01:57,900][INFO ][env ] [Jaren] using [1] data paths, mounts [[/var (/dev/mapper/VGv0-LV_var)]], net usable_space [13.7gb], net total_space [15.7gb], types [ext4]
[2015-07-14 12:01:57,934][INFO ][watcher.trigger.schedule ] [Jaren] using [ticker] schedule trigger engine
[2015-07-14 12:02:00,415][INFO ][node ] [Jaren] initialized
[2015-07-14 12:02:00,415][INFO ][node ] [Jaren] starting ...
[2015-07-14 12:02:00,470][INFO ][transport ] [Jaren] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/111.111.111.111:9300]}
[2015-07-14 12:02:00,480][INFO ][discovery ] [Jaren] elasticsearch/bZneklHbTua7zPeRRD69uQ
[2015-07-14 12:02:04,247][INFO ][cluster.service ] [Jaren] new_master [Jaren][bZneklHbTua7zPeRRD69uQ][server1][inet[/111.111.111.111:9300]], reason: zen-disco-join (elected_as_master)
[2015-07-14 12:02:04,300][INFO ][http ] [Jaren] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/111.111.111.111:9200]}
[2015-07-14 12:02:04,300][INFO ][node ] [Jaren] started
[2015-07-14 12:02:04,319][INFO ][watcher ] [Jaren] starting watch service...
[2015-07-14 12:02:04,323][INFO ][license.plugin.core ] [Jaren] license for [watcher] - valid
[2015-07-14 12:02:04,337][ERROR][watcher.license ] [Jaren]

Watcher license will expire on [Thursday, August 13, 2015]. All configured actions on

all registered watches are throttled (not executed) on Watcher license expiration.

Watches will continue be evaluated and watch history will continue being recorded.

Have a new license? please update it. Otherwise, please reach out to your support contact.

[2015-07-14 12:02:04,362][INFO ][gateway ] [Jaren] recovered [3] indices into cluster_state
[2015-07-14 12:02:04,952][INFO ][watcher ] [Jaren] watch service has started

Does this help?

If i remove watcher and the license and restart, the feed comes back again.

[dmccuk (Dennis)]

oh, by the way this is a single node cluster. I've tried restarting logstash. I get this message repeating in the logstash.log:

log4j, [2015-07-14T12:11:20.218] WARN: org.elasticsearch.transport.netty: [server1-9243-4232] Message not fully read (request) for [191] and action [discovery/zen/join/validate], resetting
log4j, [2015-07-14T12:11:23.227] WARN: org.elasticsearch.transport.netty: [server1-9243-4232] Message not fully read (request) for [192] and action [discovery/zen/join/validate], resetting

Regards

[dmccuk (Dennis)]

Actually I think I fixed it. Checking that alert pushed me to find this post:

After install watcher logstash cannot join the cluster

Hello good people, I have 3 nodes cluster and 2 logstash instance use elasticsearch node output. After installing watcher for all 3 nodes, the logstash instances repeatedly showing the following warning and cannot do anything more. WARN: org.elasticsearch.transport.netty: [logstash-*******5-6181-2044] Message not fully read (request) for [4860] and action [discovery/zen/join/validate], resetting INFO: org.elasticsearch.discovery.zen: [logstash-*******5-6181-2044] failed to send join request t…

I updated my output protocol and it kicked back into life. Kibana3 still wasn't showing me the output so i change this line back to port 9200 and the data appeared.

elasticsearch: "http://"+window.location.hostname+":9200",

Hopefully i can get on with testing watcher now.

Regards

Dennis

[mvg (Martijn Van Groningen)]

Correct, elasticsearch output should be used. Happy testing