The confusion of configuring cluster.initial_master_nodes

[yc1024]

Hi
When I read the modules-discovery-bootstrap-cluster documentation, I found the following description

When you start a master-eligible node, you can provide this setting on the command line or in the elasticsearch.yml file. After the cluster has formed, this setting is no longer required. It should not be set for master-ineligible nodes, master-eligible nodes joining an existing cluster, or cluster restarts.

What happens if I configure the cluster.initial_master_nodes parameter on master-ineligible nodes, master-eligible nodes joining an existing cluster, or cluster restarts?

[DavidTurner]

Usually nothing too bad happens, but if a number of other things go wrong and this setting is set then you might accidentally bootstrap another cluster and lose data as a result.

[yc1024]

@DavidTurner Thanks for your replay.

I have a newly installed cluster with 3 masters and 3 nodes. I configured this parameter on all the master and nodes. At startup, 2 masters and 2 nodes form a cluster, and another master and node cannot join the cluster.

The error message is as follows：

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid dxsFw3OlT9izvLFXvAGBAw than local cluster uuid KjQI8jWoRUusEg9NEhXtFg, rejecting

I think this problem is caused by incorrect cluster.initial_master_nodes parameter configuration.Is my understanding correct?

[DavidTurner]

Sort of -- this happens if you started the nodes without setting cluster.initial_master_nodes (or having set it incorrectly) and then started them again having set cluster.initial_master_nodes correctly. The manual contains more details.

[yc1024]

@DavidTurner Thanks again. The situation I encountered is different from what you described, I will add some information now.

It is the first time i start the cluster,i have configed discovery.seed_hosts and cluster.initial_master_nodes as follow，I configured these two configuration items on all master and node, and never modified.

discovery.seed_hosts: ipOfMaster1：9300,ipOfMaster2：9300,ipOfMaster3：9300
cluster.initial_master_nodes:nameOfMaster1,nameOfMaster2,nameOfMaster3

The key logs are as follows:

Node1:
[2020-04-22T15:06:25,345][INFO ][main][o.e.n.Node               ] [Node1] starting ...
[2020-04-22T15:06:26,373][INFO ][main][o.e.c.c.Coordinator      ] [Node1] cluster UUID [KjQI8jWoRUusEg9NEhXtFg]
[2020-04-22T15:06:27,830][INFO ][elasticsearch[Node1][transport_worker][T#2]][o.e.c.c.JoinHelper       ] [Node1] failed to join {Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{m} with JoinRequest{sourceNode={Node1}{nj0tbhakR32h5NAT3ZCJ5Q}{G-_NanniR-Ww4iOJBCXdGg}{{d}, optionalJoin=Optional.empty}
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid dxsFw3OlT9izvLFXvAGBAw than local cluster uuid KjQI8jWoRUusEg9NEhXtFg, rejecting

Master1:
[2020-04-22T15:06:24,300][INFO ][main][o.e.d.DiscoveryModule    ] [Master1] using discovery type [zen] and seed hosts providers [settings]
[2020-04-22T15:06:25,305][INFO ][main][o.e.n.Node               ] [Master1] starting ...
[2020-04-22T15:06:26,419][INFO ][main][o.e.c.c.Coordinator      ] [Master1] cluster UUID [KjQI8jWoRUusEg9NEhXtFg]
[2020-04-22T15:06:27,536][INFO ][elasticsearch[Master1][transport_worker][T#14]][o.e.c.c.JoinHelper       ] [Master1] failed to join {Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m} with JoinRequest{sourceNode={Master1}{zp_Ep4ADSAS0EQnMLwCkoA}{3QCCQEV6QyOHBfIXykwJEA}{}{}{m}, optionalJoin=Optional.empty}
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid dxsFw3OlT9izvLFXvAGBAw than local cluster uuid KjQI8jWoRUusEg9NEhXtFg, rejecting

Master2:
[2020-04-22T15:06:24,255][INFO ][main][o.e.d.DiscoveryModule    ] [Master2] using discovery type [zen] and seed hosts providers [settings]
[2020-04-22T15:06:25,270][INFO ][main][o.e.n.Node               ] [Master2] starting ...
[2020-04-22T15:06:26,979][INFO ][elasticsearch[Master2][generic][T#4]][o.e.c.c.Coordinator      ] [Master2] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-Master1,7mv14SBoSvmNkgmi2FYQFQ,S4WT_EGjQri1PFaPpundsg}
[2020-04-22T15:06:27,345][INFO ][elasticsearch[Master2][generic][T#4]][o.e.c.c.CoordinationState] [Master2] cluster UUID set to [dxsFw3OlT9izvLFXvAGBAw]
[2020-04-22T15:06:27,452][INFO ][elasticsearch[Master2][clusterApplierService#updateTask][T#1]][o.e.c.s.ClusterApplierService] 
[Master2] master node changed {previous [], current [{Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{ip}{ip:port}{m}]}, 
added {{Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m}}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m}}

Master3:
[2020-04-22T15:06:20,175][INFO ][main][o.e.n.Node               ] [Master3] starting ...
[2020-04-22T15:06:27,031][INFO ][elasticsearch[EsMaster@10.162.146.158][masterService#updateTask][T#1]][o.e.c.s.MasterService    ] [Master3] elected-as-master ([2] nodes joined)[Master2}{S4WT_EGjQri1PFaPpundsg}{lRib4l65SuCWTCwxqNampg}{}{}{m} elect leader,{Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m}]},added {{Master2}{S4WT_EGjQri1PFaPpundsg}{lRib4l65SuCWTCwxqNampg}{}{}{m}}

[DavidTurner]

Are you sure that all of these nodes have storage that persists across restarts?

[yc1024]

@DavidTurner I am sure that all of these nodes have storage that persists across restarts.
For each master and node, I have mounted a disk in a different path.

[DavidTurner]

In which case you have definitely configured cluster.initial_master_nodes differently at some point in their pasts. It's easily done, and the fix is described in the link I shared above.

[yc1024]

Unfortunately，i haven't modified it since I configured cluster.initial_master_nodes and discovery.seed_hosts, and this is the first time I started the newly installed cluster.
This is the second time I have encountered this problem.

I once asked such a question：

Why node uuid replace by {bootstrap-placeholder}-{NodeName}

At the time the cluster bootstrapped, it did not know the ID of master 3 so it uses a placeholder that will be replaced with master 3's ID later. This is quite a low-level implementation detail that almost certainly isn't relevant to your problem You did not set cluster.initial_master_nodes to the same value on all three nodes, or you have changed it after the first time you started these nodes. Not obviously, but if you changed cluster.initial_master_nodes then that was wrong.

[DavidTurner]

@yc1024

this is the first time I started the newly installed cluster

That is not consistent with the logs you shared. These lines indicate that this is not the first time these nodes were started:

[2020-04-22T15:06:26,373][INFO ][main][o.e.c.c.Coordinator      ] [Node1] cluster UUID [KjQI8jWoRUusEg9NEhXtFg]
[2020-04-22T15:06:26,419][INFO ][main][o.e.c.c.Coordinator      ] [Master1] cluster UUID [KjQI8jWoRUusEg9NEhXtFg]

Specifically, the first time a node starts it logs a message saying cluster UUID set to [...] but the vital words set to are missing in these logs.

[yc1024]

When the node is started for the first time, the log in cluster UUID set to [...] format must be printed?
When will the log like cluster UUID [...] be printed?

[DavidTurner]

@yc1024

When will the log like cluster UUID [...] be printed?

That's logged when the node restarts having already joined a cluster.