Elastic 2.3.4 Client nodes do not see each other

Elastic Stack Elasticsearch
1

I am having a problem with Elastic 2.3.4 where nodes do not see each other.
When I try to boot them, both nodes are "started", but when checking cluster health I get master_not_discovered_exception (503)

my configurations are:

cluster.name:my_cluster
node.name: node_${HOSTNAME}
transport.tcp.port: 9300
http.port: 9200
discovery.zen.minimum_master_nodes:2
discovery.zen.ping.unicast.hosts: ["host1", "host2"]
gateway.recover_after_nodes: 1
gateway.expected_nodes: 2

When I change discovery.zen.minimum_master_nodes to 1 (just for the test), each node work as master (status=green), but they do not see the other:

delaying initial state recovery for [5m]. expecting [2] nodes, but only have [1]

How can I fix it, please?

2

Hi @Moshe_Sucaz,

I don't see network.host (see reference docs) and by default each instance binds only to loopback. Are both instances on different machines?

I assume this is just your test setup but please be aware that 2 nodes is not ideal because with discovery.zen.minimum_master_nodes = 1 you risk split brain and with discovery.zen.minimum_master_nodes= 2 you reduce availability (as both master nodes need to be up all the time).

Daniel

3

Thanks Daniel,
In our 1.5.2 production cluster we have more then 30 nodes, each on different machine. Now I am upgrading to 2.3.4, so I just wanted to test the new setup. Also in the test env, both instances on different machines.
in 1.5.2 our setting are:

cluster.name:my_cluster
node.name: node_${HOSTNAME}
node.tag: gen1
transport.tcp.port: 9300
http.port: 9200
http.cors.allow-origin:
gateway.recover_after_nodes: 11
gateway.expected_nodes: 25
discovery.zen.minimum_master_nodes: 11

Please note that network.host is not defined in 1.5.2 and its working fine. So, which setup do you suggest me to set in 2.3.4, and keep same configuration file for all, as in 1.5.2? (I understand that if I set network.host to a fixed IP address, I can't use same configuration file).
Also, I can run my test on 8 new machines that I have, if needed...
Thanks!

4

It worked after setting network.host to ${HOSTNAME}.
(and discovery.zen.minimum_master_nodes= 2)
In that way I could still use the same configuration file to all.
Is it OK?

5

Hi @Moshe_Sucaz,

seems fine, just keep our recommendation re. discovery.zen.minimum_master_nodes in mind:

Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1)

Daniel

6

Thanks!