Timeout/Connection error


(Matthew D Johnson) #1

I have an error that I hope I some one in the group can help. In my client log I get the following error:

_Caused by: org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$4.onTimeout(TransportMasterNodeOperationAction.java:170) ~[na:na]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239) ~[na:na]
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:494) ~[na:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]

In my elasticsearch node I have the following error:

[2016-10-20 00:02:36,144][WARN ][cluster.service ] [Sin] failed to reconnect to node [prd-use1b-pr-15-epsapp-03x01x00-0009.prv-openclass.com][SxvqPTfzR8uUvlsCPFGAww][prd-use1b-pr-15-epsapp-03x01x00-0009.prv-openclass.com][inet[/127.0.1.1:9300]]{data=false, client=true, zone=use1b}
org.elasticsearch.transport.ConnectTransportException: [prd-use1b-pr-15-epsapp-03x01x00-0009.prv-openclass.com][inet[/127.0.1.1:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:656)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:624)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:146)
at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:518)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /127.0.1.1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
~

My configuration looks like the following:

cluster.name: epselastic
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: use1b,use1c,use1d

node.zone: use1b

path.data: /mnt/es

network.host: xxx.xxx.xx.67

discovery:
type: com.sonian.elasticsearch.zookeeper.discovery.ZooKeeperDiscoveryModule
sonian.elasticsearch.zookeeper:
client.host: prd-use1b-pr-15-epszookeeper-1x0x1083-0001.prv-openclass.com:2181,prd-use1c-pr-15-epszookeeper-1x0x1083-0001.prv-openclass.com:2181,prd-use1d-pr-15-epszookeeper-1x0x1083-0001.prv-openclass.com:2181

script.disable_dynamic: false

I am not sure what is causing this error. Thanks in advance for any help.


(Zachary Tong) #2

I'm not sure we can really provide any help, since you're using the Zookeeper plugin. Basically, the error says the node couldn't find a new master node to talk to. Master detection and discovery are normally controlled via the internal Zen service, but the Zookeeper plugin replaces it with Zookeeper.

Replacing something as large/complicated as discovery in Elasticsearch with an external service is bound to be difficult and/or buggy. It may have been a transient network blip, or it may have been some bug in the unofficial plugin. It's hard to say since we don't maintain it.

I'd say open a ticket with the maintainers of the plugin, or revert back to the regular Zen discovery service.


Can't connect to node
(Matthew D Johnson) #3

Thanks for the prompt replied


(system) #4