After install watcher logstash cannot join the cluster

Elastic Stack Elasticsearch
1

Hello good people,

I have 3 nodes cluster and 2 logstash instance use elasticsearch node output. After installing watcher for all 3 nodes, the logstash instances repeatedly showing the following warning and cannot do anything more.

WARN: org.elasticsearch.transport.netty: [logstash-*******5-6181-2044] Message not fully read (request) for [4860] and action [discovery/zen/join/validate], resetting

INFO: org.elasticsearch.discovery.zen: [logstash-*******5-6181-2044] failed to send join request to master [[********5][YdGHuHyFTWiHNunW1QusOQ][********5][inet[/***.***.***.**5:9300]]{master=true}], reason [org.elasticsearch.transport.RemoteTransportException: [*********5][inet[/***.***.***.**5:9300]][discovery/zen/join]; org.elasticsearch.transport.RemoteTransportException: [logstash-*********5-6181-2044][inet[/***.***.***.**5:9301]][discovery/zen/join/validate]; org.elasticsearch.ElasticsearchIllegalArgumentException: No custom index metadata factory registered for type [licenses]]

I can see something wrong with the license, might be related to installing elasticsearch/license/latest. I think I can resolve this by uninstalling the license plugin but I cannot test watcher in this case. And in order to let logstash functional normally I might change to elasticsearch-http output for a while.

Any idea about this?

Thanks a lot!

2

The elasticsearch-http output is deprecated, use the elasticsearch output with the http protocol.

However this is interesting, I will ask one of the watcher dev team to take a look :slight_smile:

3

This seems to be an error because Logstash attempts to join the cluster with a NodeClient and fails because it doesn't have the Elasticsearch license plugin installed.

Possible workarounds include not using the default protocol (node), and using either transport or http:

output {
  elasticsearch {
    protocol => http
    ...
  }
}
4

Yeah, I am already using the http protocol in elasticsearch output to let Logstash functional normally. However, this would stop me monitoring the health of Logstash instances by Node Stats API.