Can't form a cluster with more than one node ".deb" debian 9 : elasticsearch 7.2

Hello everyone,

I want to form a cluster with 3 nodes, but i have some errors.
Here's my configuration:

so i have 3 debian vm, connected in Bridged mode <net: 10.2.1.0/24>

  • my /etc/hosts file node-1

127.0.0.1 localhost
127.0.1.1 elasticsearch-node-1

10.2.1.70 elastic-node-1
10.2.1.150 elastic-node-2
10.2.1.90 elastic-node-3

::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

  • my /etc/hosts file node-2

127.0.0.1 localhost
127.0.1.1 elasticsearch-node-2

10.2.1.70 elastic-node-1
10.2.1.150 elastic-node-2
10.2.1.90 elastic-node-3

::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

  • my /etc/hosts file node-3

127.0.0.1 localhost
127.0.1.1 elasticsearch-node-3

10.2.1.70 elastic-node-1
10.2.1.150 elastic-node-2
10.2.1.90 elastic-node-3

::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

  • my conf file of elasticsearch node [elastic-node-1/elastic-node-2/elastic-node-3]
# ---------------------------------- Cluster -----------------------------------
cluster.name: my-application
#------------------------------------ Node ------------------------------------
node.master: true
node.data: true
# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# ----------------------------------- Memory -----------------------------------
#bootstrap.memory_lock: true
# ---------------------------------- Network -----------------------------------
network.host: _site_
#http.port: 9200
# --------------------------------- Discovery ----------------------------------
cluster.initial_master_nodes: ["elastic-node-1", "elastic-node-2", "elastic-node-3"]
discovery.seed_hosts: ["elastic-node-1", "elastic-node-2", "elastic-node-3"]

when i start elasticsearch with this configuration, each node from his one cluster, with a different UUID cluster

i have for the elastic-node-1

cluster UUID [-MpG6MjNSkuYEGJaQE_oUA]

for elastic-node-2

cluster UUID [fukun4w4QVacD3RXjAKRcQ]

for my elastic-node-3

cluster UUID [rYkVboMdRp23swjU4_nc2w]

when i try to force the elastic-node-1 to be the master and the elastic-node-2 and elastic-node-3 to be just data node with :

for the elastic-node-1

node.master: true
node.data: true

for the elastic-node-2/elastic-node-3

node.master: false
node.data: true

i have this logs errors in the elastic-node-1

[2019-06-28T18:02:27,513][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-node-1] master node changed {previous , current [{elasticsearch-node-1}{0fau9w2yR52tUYSsO8mkBg}{5vcSn7shQ9K3q2W48UD1sw}{10.2.1.70}{10.2.1.70:9300}{ml.machine_memory=2101805056, xpack.installed=true, ml.max_open_jobs=20}]}, term: 14, version: 52, reason: Publication{term=14, version=52}
[2019-06-28T18:02:27,597][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-node-1] publish_address {10.2.1.70:9200}, bound_addresses {10.2.1.70:9200}
[2019-06-28T18:02:27,598][INFO ][o.e.n.Node ] [elasticsearch-node-1] started
[2019-06-28T18:02:27,872][WARN ][o.e.c.c.Coordinator ] [elasticsearch-node-1] failed to validate incoming join request from node [{elasticsearch-node-3}{QRIywrOqS-CzcGJM_xod2Q}{8U7kOxt7Q-GB_0E0q6UPIw}{10.2.1.90}{10.2.1.90:9300}{ml.machine_memory=2101805056, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [elasticsearch-node-3][10.2.1.90:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid -MpG6MjNSkuYEGJaQE_oUA than local cluster uuid rYkVboMdRp23swjU4_nc2w, rejecting
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-06-28T18:02:28,147][WARN ][o.e.c.c.Coordinator ] [elasticsearch-node-1] failed to validate incoming join request from node [{elasticsearch-node-2}{vYHO8nrRTWmQt037S4hFTw}{C60jPO8yRbeQPlDjwfBAiA}{10.2.1.150}{10.2.1.150:9300}{ml.machine_memory=2101805056, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [elasticsearch-node-2][10.2.1.150:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid -MpG6MjNSkuYEGJaQE_oUA than local cluster uuid fukun4w4QVacD3RXjAKRcQ, rejecting
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]
...

i have this logs errors in the elastic-node-2

[2019-06-28T18:02:21,922][INFO ][o.e.b.BootstrapChecks ] [elasticsearch-node-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-28T18:02:21,985][INFO ][o.e.c.c.Coordinator ] [elasticsearch-node-2] cluster UUID [fukun4w4QVacD3RXjAKRcQ]
[2019-06-28T18:02:28,255][INFO ][o.e.c.c.JoinHelper ] [elasticsearch-node-2] failed to join {elasticsearch-node-1}{0fau9w2yR52tUYSsO8mkBg}{5vcSn7shQ9K3q2W48UD1sw}{10.2.1.70}{10.2.1.70:9300}{ml.machine_memory=2101805056, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-node-2}{vYHO8nrRTWmQt037S4hFTw}{C60jPO8yRbeQPlDjwfBAiA}{10.2.1.150}{10.2.1.150:9300}{ml.machine_memory=2101805056, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=14, lastAcceptedTerm=9, lastAcceptedVersion=39, sourceNode={elasticsearch-node-2}{vYHO8nrRTWmQt037S4hFTw}{C60jPO8yRbeQPlDjwfBAiA}{10.2.1.150}{10.2.1.150:9300}{ml.machine_memory=2101805056, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-node-1}{0fau9w2yR52tUYSsO8mkBg}{5vcSn7shQ9K3q2W48UD1sw}{10.2.1.70}{10.2.1.70:9300}{ml.machine_memory=2101805056, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [elasticsearch-node-1][10.2.1.70:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]
...

What should i do to fix it ?? <start my cluster with my 3 nodes>

Thank you.

I think this note in the docs describes the situation you're in, and also the solution.

1 Like

Thank you @DavidTurner work for me :smiley:,

NB : I had to do a "chown -R elasticsearch:elasticsearch /new datapath/elasticsearch" in addition to make it work.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.