Failed to send join request to master, no known master node

Elastic Stack Elasticsearch
1

Hi i have a cluster with 3 nodes and work perfectly since 3 months, today i have found the node 3 disconnect, i have restart elasticsearch without effect, the error is:

[2016-09-07 14:38:59,297][INFO ][node                     ] [es-node3] starting ...
[2016-09-07 14:38:59,393][INFO ][transport                ] [es-node3] publish_address {ec2-52-208-xxx-11.eu-west-1.compute.amazonaws.com/10.1.200.190:9300}, bound_addresses {[::]:9300}
[2016-09-07 14:38:59,398][INFO ][discovery                ] [es-node3] myz-es-cluster/RSBMKUfLQiCDo8zWLFDmWQ
[2016-09-07 14:39:29,401][WARN ][discovery                ] [es-node3] waited for 30s and no initial state was set by the discovery
[2016-09-07 14:39:29,417][INFO ][http                     ] [es-node3] publish_address {ec2-52-208-xxx-11.eu-west-1.compute.amazonaws.com/10.1.200.190:9200}, bound_addresses {[::]:9200}
[2016-09-07 14:39:29,418][INFO ][node                     ] [es-node3] started
[2016-09-07 14:39:35,084][DEBUG][action.admin.indices.create] [es-node3] no known master node, scheduling a retry
[2016-09-07 14:39:35,135][DEBUG][action.admin.indices.create] [es-node3] no known master node, scheduling a retry
[2016-09-07 14:40:02,448][INFO ][discovery.zen            ] [es-node3] failed to send join request to master [{es-node2}{1aFV4V3DRd65BMJJ_BhU1Q}{10.1.200.10}{10.1.200.10:9300}{master=true}], reason [ElasticsearchTimeoutException[Timeout waiting for task.]]
[2016-09-07 14:40:09,390][DEBUG][action.admin.indices.create] [es-node3] no known master node, scheduling a retry
[2016-09-07 14:40:09,390][DEBUG][action.admin.indices.create] [es-node3] no known master node, scheduling a retry

Node 1 and node 2 work, actually node 2 is a master, the config file of node 3 is:

cluster.name: es-cluster
node.name: es-node3
bootstrap.mlockall: true
discovery.zen.ping.unicast.hosts: ["ec2-52-208-xxx-11.eu-west-1.compute.amazonaws.com", "ec2-52-51-160-xxx.eu-west-1.compute.amazonaws.com", "ec2-52-208-xxx-163.eu-west-1.compute.amazonaws.com"]
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.multicast.enabled: false
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: ec2-52-208-xxx-11.eu-west-1.compute.amazonaws.com

This config is the same for all nodes, only "network.publish_host" is different, all node is in the same lan without firewall

2

Hey,

without more information, this looks more like a network/firewall problem. Can you reach the other nodes on port 9300 from that node (e.g. using telnet or netcat)? If that does not work, it is more likely a configuration issue.

--Alex