Failed to connect to master


#1

Hi,

Our logstash(indexer) recently encountered below error frequently, what is this error and how can we fix it?

Nov 03, 2015 2:00:25 PM org.elasticsearch.discovery.zen.ZenDiscovery joinElectedMaster
WARNING: [logstash-ELKServerName-24033-11634] failed to connect to master [[ELKServerName][an-1G0IpRja1urW2OD4jxw]...
org.elasticsearch.transport.ConnectTransportException: [ELKServerName][inet[/ELKServerIP:9300]] connect_timeout[
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:825)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:758)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:731)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:216)
at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:465)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:417)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$6100(ZenDiscovery.java:86)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1384)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

redis, logstash(indexer) and elasticsearch run on the same server, and it's a single node cluster.
logstash: 1.5.2
elasticsearch:1.7.2


(Mark Walkom) #2

Try switching to the HTTP protocol and you should be ok :slight_smile:


#3

are you referring to configuring indexer's elasticsearch output plugin to use protocol => "http"?


(Mark Walkom) #4

Yep! :slight_smile:


#5

thank you very much! let me try it~~


#6

I still encounter the same error after changing to http protocol :frowning:
output {
elasticsearch {
cluster => "clustername"
protocol => "http"
index => "test-%{+YYYY.MM.dd}"
workers => 4
}
}


(Mark Walkom) #7

You need to remove cluster and put in host, check out https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html


#8

I tried with below option but still failed. with the doc, i'm not sure whether i need to configure the host to host => "ip:9200", but anyway i tried both options, but both still came back with errors
output {
elasticsearch {
protocol => "http"
host => "xxx.xxx.xxx.xxx"
index => "test-%{+YYYY.MM.dd}"
workers => 4
}
}


(Magnus B├Ąck) #9

I tried with below option but still failed

How? If you made the configuration change you claimed to have made you must have received a different error message.


#10

i've actually compared the error message, and to my surprise, they are the same.
output {
elasticsearch {
protocol => "http"
host => "xxx.xxx.xxx.xxx"
index => "test-%{+YYYY.MM.dd}"
workers => 4
}
}

output {
elasticsearch {
protocol => "http"
host => "xxx.xxx.xxx.xxx:9200"
index => "test-%{+YYYY.MM.dd}"
workers => 4
}
}


(Mark Walkom) #11

Then your config is wrong, can we see the entire thing?


#12

which part of the configuration you would like to see?
the interesting thing is that the whole log ingestion was working fine for about 2 weeks, then the above error started to surface when we added in more logs along the way


(Mark Walkom) #13

The whole thing would help.


(system) #14