I configured two nodes on two Linux VMs and it is working. Currently trying to setup a third node on the cluster in my local windows machine. Getting the errors below
[node-3] failed to join {node-1}{3eMmu1kVQtuB95A1KX4cXQ}{2m4B33pgQPCN7B6oeZuaww}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}{ml.machine_memory=16599810048, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=8300527616, transform.node=true} with JoinRequest{sourceNode={node-3}{gxNAxl-xQqirbIpQ30NXYQ}{9UhA-NSxTyqtcA0HbjAGzA}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}{ml.machine_memory=8442609664, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1037959168}, minimumTerm=6, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [node-1][**.***.***.***:9300][internal:cluster/coordination/join]
master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [**.***.***.***:9300] to bootstrap a cluster
````````````````````````````````````````````````````````
You have set cluster.initial_master_nodes but the docs say:
IMPORTANT: After the cluster forms successfully for the first time, remove the cluster.initial_master_nodes setting from each nodes' configuration. Do not use this setting when restarting a cluster or adding a new node to an existing cluster.
I commented out the cluster.initial_master_nodes from all the three nodes and tried running the windows machine node again. Now it says [cluster.initial_master_nodes] is empty on this node.
[node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{gxNAxl-xQqirbIpQ30NXYQ}{ZPvpYSMJS2qZ_jLOXubBpA}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}, {node-1}{y2WpONChR_WD3DgQYT1VXg}{dfY9UqqeRViIX3xv6IJuTQ}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}, {node-2}{L9EX348MSrypvQDxuTFMLQ}{uXev5N_FR7qGquhjlblZpw}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}]; discovery will continue using [**.***.***.***:9300, **.***.***.***:9300] from hosts providers and [{node-3}{gxNAxl-xQqirbIpQ30NXYQ}{ZPvpYSMJS2qZ_jLOXubBpA}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}] from last-known cluster state; node term 6, last-accepted version 0 in term 0
[2021-06-14T21:29:40,121][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{gxNAxl-xQqirbIpQ30NXYQ}{ZPvpYSMJS2qZ_jLOXubBpA}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}, {node-1}{y2WpONChR_WD3DgQYT1VXg}{dfY9UqqeRViIX3xv6IJuTQ}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}, {node-2}{L9EX348MSrypvQDxuTFMLQ}{uXev5N_FR7qGquhjlblZpw}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}]; discovery will continue using [**.***.***.***:9300, **.***.***.***:9300] from hosts providers and [{node-3}{gxNAxl-xQqirbIpQ30NXYQ}{ZPvpYSMJS2qZ_jLOXubBpA}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}] from last-known cluster state; node term 6, last-accepted version 0 in term 0
[2021-06-14T21:29:43,967][WARN ][o.e.t.TransportService ] [node-3] Received response for a request that has timed out, sent [3.3s/3317ms] ago, timed out [210ms/210ms] ago, action [internal:discovery/request_peers], node [{node-2}{L9EX348MSrypvQDxuTFMLQ}{uXev5N_FR7qGquhjlblZpw}{**.***.***.***}{**.***.***.***:9300}{cdfhilmrstw}{ml.machine_memory=16599810048, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=8300527616, transform.node=true}], id [91]
It's hard to tell what else is wrong from the few log messages you've shared. I think there will be other log messages that are more helpful. Received response for a request that has timed out indicates a timeout, maybe faulty network or maybe the nodes are overloaded.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.