Node failed to send join request to master

Elastic Stack Elasticsearch
1

Hi,

I have an elasticsearch cluster with 4 nodes (4 windows hosts, each running an ubuntu VM where elasticsearch is running) currently running smoothly without any issues. I am trying to add node 5 to the cluster, but am getting an error.

Here is the error from the master node.

I've tried deleting the nodes directory, increasing the minimum number of master nodes from 2 to 3, and just recloning the VM. No luck.

Any advice on how to solve this problem?

Thanks!

2

The connection attempt timed out:

IOException[connection timed out: 192.168.10.24/192.168.10.24:9300]

This normally indicates a connectivity issue.

The log you quote from the master is dated two days earlier than the other logs and therefore doesn't seem related.

3

Thanks for the response.

I thought it was a connectivity issue, but am not sure how to go about troubleshooting it further. The log timing is off because I haven't really bothered to sync the clocks on the VMs, but I can confirm that the log on the master appears when I try to connect the node to the cluster.

I am able to ping to the other nodes in the cluster from the node_5, so I'm not sure what exactly is happening. Any more help would be appreciated.

4

Not related, but if you have 4 master-eligible nodes then you must set minimum_master_nodes to 3.

Possibly unrelated, but you should know that Elasticsearch performs some optimisations that assume the nodes' clocks are reasonably in sync. It does the right thing even if they're not closely synchronised, but potentially more slowly and/or using more memory.

I note that you're using TLS (because o.e.x.s.t.n.SecurityNetty4ServerTransport). It's possible that TLS requires your clocks to be more accurately synchronised, although if it is this then I think we should report it more loudly.

ping is rather a different thing from the TCP connections that Elasticsearch expects to be able to open. Try nc or telnet. If you can establish basic connectivity using these tools, try starting the new node with this in its elasticsearch.yml file:

logger.org.elasticsearch.transport:TRACE

This should give much more verbose logs.

5

After trying out your suggestions, I realised that this node had Symantec installed (but the other nodes do not) so I just had to configure that and the node joined the cluster. Reminder for myself not to assume that all my nodes are necessarily exactly the same. Thanks!

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.