Failed to connect to master

Elastic Stack Logstash
1

Hi,

Our logstash(indexer) recently encountered below error frequently, what is this error and how can we fix it?

Nov 03, 2015 2:00:25 PM org.elasticsearch.discovery.zen.ZenDiscovery joinElectedMaster
WARNING: [logstash-ELKServerName-24033-11634] failed to connect to master [[ELKServerName][an-1G0IpRja1urW2OD4jxw]...
org.elasticsearch.transport.ConnectTransportException: [ELKServerName][inet[/ELKServerIP:9300]] connect_timeout[
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:825)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:758)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:731)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:216)
at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:465)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:417)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$6100(ZenDiscovery.java:86)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1384)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

redis, logstash(indexer) and elasticsearch run on the same server, and it's a single node cluster.
logstash: 1.5.2
elasticsearch:1.7.2

2

Try switching to the HTTP protocol and you should be ok :slight_smile:

3

are you referring to configuring indexer's elasticsearch output plugin to use protocol => "http"?

4

Yep! :slight_smile:

5

thank you very much! let me try it~~

6

I still encounter the same error after changing to http protocol :frowning:
output {
elasticsearch {
cluster => "clustername"
protocol => "http"
index => "test-%{+YYYY.MM.dd}"
workers => 4
}
}

7

You need to remove cluster and put in host, check out https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html

8

I tried with below option but still failed. with the doc, i'm not sure whether i need to configure the host to host => "ip:9200", but anyway i tried both options, but both still came back with errors
output {
elasticsearch {
protocol => "http"
host => "xxx.xxx.xxx.xxx"
index => "test-%{+YYYY.MM.dd}"
workers => 4
}
}

9

I tried with below option but still failed

How? If you made the configuration change you claimed to have made you must have received a different error message.

10

i've actually compared the error message, and to my surprise, they are the same.
output {
elasticsearch {
protocol => "http"
host => "xxx.xxx.xxx.xxx"
index => "test-%{+YYYY.MM.dd}"
workers => 4
}
}

output {
elasticsearch {
protocol => "http"
host => "xxx.xxx.xxx.xxx:9200"
index => "test-%{+YYYY.MM.dd}"
workers => 4
}
}

11

Then your config is wrong, can we see the entire thing?

12

which part of the configuration you would like to see?
the interesting thing is that the whole log ingestion was working fine for about 2 weeks, then the above error started to surface when we added in more logs along the way

13

The whole thing would help.