Failed to reconnect to node


Oftentimes, in my cluster, elasticsearch data nodes are trying to reach the elasticsearch master node using an IP addresses different from the one defined in master node's elasticsearch.yml with the statement:

They are actually trying to reach the IP addresses the master node uses for its own iSCSI devices.
Why is this happening?

As stated in elasticsearch.yml comments, I expected to set both network.bind_host and network.publish_host, with network.publish_host being the address other nodes will use to communicate with this node.

Following, the logging of one data node of mine.
Note that is the IP address my master node has configured on the iSCSI subnet.

[2015-10-01 13:31:45,174][WARN ][cluster.service          ] [**DataNode1**] failed to reconnect to node [logstash-**MasterNode**-2287-13610][_XtwgM4ER0OBiPZdCyTS8g][**MasterNode**][inet[/]]{client=true, data=false}
org.elasticsearch.transport.ConnectTransportException: [logstash-**MasterNode**-2287-13610][inet[/]] connect_timeout[30s]
	at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
	at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
	at org.elasticsearch.transport.netty.NettyTransport.connectToNode(
	at org.elasticsearch.transport.TransportService.connectToNode(
	at org.elasticsearch.cluster.service.InternalClusterService$
	at java.util.concurrent.ThreadPoolExecutor.runWorker(
	at java.util.concurrent.ThreadPoolExecutor$
Caused by: Connection refused: /
	at Method)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$
	... 3 more

Thank you so much,


Well, after some troubleshooting I found out the problem. I didn't know logstash partecipated to the elasticsearch cluster, I realized it seeing how it opened a socket on the unexpected addresses and port 9301. So I found out that the statement

host => "ipaddress"

, in the elasticsearch plugin of the logstash output filter, did not bind logstash to that ip address. To bind it you must use the

bind_host => "ipaddress"

Adding this statement with the expected ip address now I don't see the exception anymore.


(system) #3