# Why cluster UUID is different

**URL:** <https://discuss.elastic.co/t/why-cluster-uuid-is-different/259660>\
**Category:** Elasticsearch\
**Created:** [December 26, 2020, 12:07pm UTC](https://discuss.elastic.co/t/why-cluster-uuid-is-different/259660 "2020-12-26T12:07:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sagittar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagittar/32/81423_2.png) [@sagittar](https://discuss.elastic.co/u/sagittar)\
**Post date:** [December 26, 2020, 12:07pm UTC](https://discuss.elastic.co/t/why-cluster-uuid-is-different/259660/1 "2020-12-26T12:07:19Z")

</div>

node1 [node\_2409:8086:8110:3f0::2d]  
node2 [node\_2409:8086:8110:3f0::2e]  
node3 [node\_2409:8086:8110:3f0::2f]

here is node1 log

[2020-12-23T02:38:57,563][INFO][o.e.c.c.JoinHelper] [node\_2409:8086:8110:3f0::2d] failed to join {node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim} with JoinRequest{sourceNode={node\_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{HdogZVSKQtu2V61zBCdNeQ}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}, optionalJoin=Optional[Join{term=19, lastAcceptedTerm=13, lastAcceptedVersion=46097, sourceNode={node\_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{HdogZVSKQtu2V61zBCdNeQ}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}, targetNode={node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}}]}  
org.elasticsearch.transport.RemoteTransportException: [node\_2409:8086:8110:3f0::2e][[2409:8086:8110:3f0::2e]:9300][internal:cluster/coordination/join]  
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node  
at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:506) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:360) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.1.jar:7.3.1]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0\_242]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0\_242]  
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_242]  
Caused by: org.elasticsearch.transport.RemoteTransportException: [node\_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]  
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid \_4xxif5aQ\_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS\_aqA, rejecting  
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

node2：  
[2020-12-23T02:38:57,372][INFO][o.e.c.s.ClusterApplierService] [node\_2409:8086:8110:3f0::2e] master node changed {previous , current [{node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}]}, removed {{node\_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim},}, added {{node\_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{p6QGBla8QkuMi-6-428Zgw}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim},}, term: 19, version: 58, reason: Publication{term=19, version=58}  
[2020-12-23T02:38:57,393][INFO][o.e.c.r.DelayedAllocationService] [node\_2409:8086:8110:3f0::2e] scheduling reroute for delayed shards in [59.3s] (1 delayed shards)  
[2020-12-23T02:38:57,450][INFO][o.e.i.s.IndexShard] [node\_2409:8086:8110:3f0::2e] [i2000.mim][0] primary-replica resync completed with 0 operations  
[2020-12-23T02:38:57,561][WARN][o.e.c.c.Coordinator] [node\_2409:8086:8110:3f0::2e] failed to validate incoming join request from node [{node\_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{HdogZVSKQtu2V61zBCdNeQ}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]  
org.elasticsearch.transport.RemoteTransportException: [node\_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]  
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid \_4xxif5aQ\_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS\_aqA, rejecting  
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

[2020-12-23T02:38:58,547][WARN][o.e.c.c.Coordinator] [node\_2409:8086:8110:3f0::2e] failed to validate incoming join request from node [{node\_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{HdogZVSKQtu2V61zBCdNeQ}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]  
org.elasticsearch.transport.RemoteTransportException: [node\_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]  
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid \_4xxif5aQ\_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS\_aqA, rejecting  
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]

node3  
[2020-12-23T02:38:56,761][INFO][o.e.c.c.JoinHelper] [node\_2409:8086:8110:3f0::2f] failed to join {node\_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{p6QGBla8QkuMi-6-428Zgw}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim} with JoinRequest{sourceNode={node\_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{p6QGBla8QkuMi-6-428Zgw}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, optionalJoin=Optional[Join{term=18, lastAcceptedTerm=1, lastAcceptedVersion=57, sourceNode={node\_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{p6QGBla8QkuMi-6-428Zgw}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, targetNode={node\_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{p6QGBla8QkuMi-6-428Zgw}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}}]}  
org.elasticsearch.transport.RemoteTransportException: [node\_2409:8086:8110:3f0::2f][[2409:8086:8110:3f0::2f]:9300][internal:cluster/coordination/join]  
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower  
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]  
at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0\_242]  
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:477) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:606) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:243) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) [elasticsearch-7.3.1.jar:7.3.1]

[2020-12-23T02:38:57,131][INFO][o.e.c.s.ClusterApplierService] [node\_2409:8086:8110:3f0::2f] master node changed {previous , current [{node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}]}, added {{node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim},}, term: 19, version: 58, reason: ApplyCommitRequest{term=19, version=58, sourceNode={node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}}  
[2020-12-23T02:38:57,390][INFO][o.e.h.AbstractHttpServerTransport] [node\_2409:8086:8110:3f0::2f] publish\_address {[2409:8086:8110:3f0::2f]:9200}, bound\_addresses {[2409:8086:8110:3f0::2f]:9200}  
[2020-12-23T02:38:57,394][INFO][o.e.n.Node] [node\_2409:8086:8110:3f0::2f] started  
[2020-12-23T02:48:13,025][INFO][o.e.c.c.Coordinator] [node\_2409:8086:8110:3f0::2f] master node [{node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] failed, restarting discovery  
org.elasticsearch.transport.NodeDisconnectedException: [node\_2409:8086:8110:3f0::2e][[2409:8086:8110:3f0::2e]:9300][disconnected] disconnected  
[2020-12-23T02:48:13,030][INFO][o.e.c.s.ClusterApplierService] [node\_2409:8086:8110:3f0::2f] master node changed {previous [{node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}], current }, term: 19, version: 112, reason: becoming candidate: onLeaderFailure  
[2020-12-23T02:48:13,163][WARN][o.e.c.NodeConnectionsService] [node\_2409:8086:8110:3f0::2f] failed to connect to {node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim} (tried [1] times)  
org.elasticsearch.transport.ConnectTransportException: [node\_2409:8086:8110:3f0::2e][[2409:8086:8110:3f0::2e]:9300] connect\_exception  
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:957) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.3.1.jar:7.3.1]  
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.3.1.jar:7.3.1]  
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0\_242]  
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0\_242]  
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0\_242]  
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0\_242]  
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.3.1.jar:7.3.1]  
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]  
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]  
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]  
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]

[2020-12-23T02:48:23,028][WARN][o.e.c.c.ClusterFormationFailureHelper] [node\_2409:8086:8110:3f0::2f] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node\_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{p6QGBla8QkuMi-6-428Zgw}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node\_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{HdogZVSKQtu2V61zBCdNeQ}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2e]:9300] from hosts providers and [{node\_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX\_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}, {node\_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{p6QGBla8QkuMi-6-428Zgw}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}] from last-known cluster state; node term 19, last-accepted version 112 in term 19

---

<div class="post-metadata">

**Author:** ![sagittar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sagittar/32/81423_2.png) [@sagittar](https://discuss.elastic.co/u/sagittar)\
**Post date:** [December 27, 2020, 2:54am UTC](https://discuss.elastic.co/t/why-cluster-uuid-is-different/259660/2 "2020-12-27T02:54:31Z")

</div>

Thanks for your help

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 29, 2020, 5:58am UTC](https://discuss.elastic.co/t/why-cluster-uuid-is-different/259660/3 "2020-12-29T05:58:38Z")

</div>

What does your config look like on each node? How did you start the cluster?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 26, 2021, 5:58am UTC](https://discuss.elastic.co/t/why-cluster-uuid-is-different/259660/4 "2021-01-26T05:58:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.