I have a cluster on version 2.4.1, and some nodes cannot connect to the master.
The error log is:
[WARN ][discovery.zen ] [...] failed to connect to master [...] retrying... org.elasticsearch.transport.ConnectTransportException: [master][x.x.x.x:9310] connect_timeout[30s] at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:1002) ~[elasticsearch-2.4.1.jar:2.4.1] at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:937) ~[elasticsearch-2.4.1.jar:2.4.1] at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:911) ~[elasticsearch-2.4.1.jar:2.4.1] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:260) ~[elasticsearch-2.4.1.jar:2.4.1] at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:444) [elasticsearch-2.4.1.jar:2.4.1] at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:396) [elasticsearch-2.4.1.jar:2.4.1] at org.elasticsearch.discovery.zen.ZenDiscovery.access$4400(ZenDiscovery.java:96) [elasticsearch-2.4.1.jar:2.4.1] at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1296) [elasticsearch-2.4.1.jar:2.4.1]
The master has a low load, netstat shows that the node has several established connections to the master, and I can establishe a TCP connection with telnet with no issue, and the master closes immediately the connection if I send junk on this connection.
Many nodes are connected with no issues to the cluster, this issue only happens for some new nodes I try to add to the cluster.
What could cause this issue ? How can I investigate what happens?