# The confusion of configuring cluster.initial\_master\_nodes

**URL:** <https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213>\
**Category:** Elasticsearch\
**Created:** [April 22, 2020, 9:09am UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213 "2020-04-22T09:09:03Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [April 22, 2020, 9:09am UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/1 "2020-04-22T09:09:03Z")

</div>

Hi  
When I read the [modules-discovery-bootstrap-cluster](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/modules-discovery-bootstrap-cluster.html) documentation, I found the following description

When you start a master-eligible node, you can provide this setting on the command line or in the elasticsearch.yml file. After the cluster has formed, this setting is no longer required. **It should not be set for master-ineligible nodes, master-eligible nodes joining an existing cluster, or cluster restarts**.

What happens if I configure the `cluster.initial_master_nodes` parameter on `master-ineligible nodes`,` master-eligible nodes joining an existing cluster`, or `cluster restarts`?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 22, 2020, 12:27pm UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/2 "2020-04-22T12:27:32Z")

</div>

Usually nothing too bad happens, but if a number of other things go wrong and this setting is set then you might accidentally bootstrap another cluster and lose data as a result.

---

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [April 22, 2020, 3:05pm UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/3 "2020-04-22T15:05:20Z")

</div>

@DavidTurner Thanks for your replay.

I have a newly installed cluster with 3 masters and 3 nodes. I configured this parameter on all the master and nodes. At startup, 2 masters and 2 nodes form a cluster, and another master and node cannot join the cluster.

The error message is as follows：

```auto
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid dxsFw3OlT9izvLFXvAGBAw than local cluster uuid KjQI8jWoRUusEg9NEhXtFg, rejecting

```

I think this problem is caused by incorrect `cluster.initial_master_nodes `parameter configuration.Is my understanding correct?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 22, 2020, 4:28pm UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/4 "2020-04-22T16:28:13Z")

</div>

Sort of -- this happens if you started the nodes without setting `cluster.initial_master_nodes` (or having set it incorrectly) and then started them again having set `cluster.initial_master_nodes` correctly. The manual [contains more details](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html#modules-discovery-bootstrap-cluster-joining).

---

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [April 23, 2020, 2:48am UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/5 "2020-04-23T02:48:56Z")

</div>

@DavidTurner Thanks again. The situation I encountered is different from what you described, I will add some information now.

It is the first time i start the cluster,i have configed `discovery.seed_hosts` and `cluster.initial_master_nodes` as follow，I configured these two configuration items on all master and node, and never modified.

```auto
discovery.seed_hosts: ipOfMaster1：9300,ipOfMaster2：9300,ipOfMaster3：9300
cluster.initial_master_nodes:nameOfMaster1,nameOfMaster2,nameOfMaster3

```

The key logs are as follows:

```auto
Node1:
[2020-04-22T15:06:25,345][INFO][main][o.e.n.Node] [Node1] starting ...
[2020-04-22T15:06:26,373][INFO][main][o.e.c.c.Coordinator] [Node1] cluster UUID [KjQI8jWoRUusEg9NEhXtFg]
[2020-04-22T15:06:27,830][INFO][elasticsearch[Node1][transport_worker][T#2]][o.e.c.c.JoinHelper] [Node1] failed to join {Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{m} with JoinRequest{sourceNode={Node1}{nj0tbhakR32h5NAT3ZCJ5Q}{G-_NanniR-Ww4iOJBCXdGg}{{d}, optionalJoin=Optional.empty}
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid dxsFw3OlT9izvLFXvAGBAw than local cluster uuid KjQI8jWoRUusEg9NEhXtFg, rejecting

Master1:
[2020-04-22T15:06:24,300][INFO][main][o.e.d.DiscoveryModule] [Master1] using discovery type [zen] and seed hosts providers [settings]
[2020-04-22T15:06:25,305][INFO][main][o.e.n.Node] [Master1] starting ...
[2020-04-22T15:06:26,419][INFO][main][o.e.c.c.Coordinator] [Master1] cluster UUID [KjQI8jWoRUusEg9NEhXtFg]
[2020-04-22T15:06:27,536][INFO][elasticsearch[Master1][transport_worker][T#14]][o.e.c.c.JoinHelper] [Master1] failed to join {Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m} with JoinRequest{sourceNode={Master1}{zp_Ep4ADSAS0EQnMLwCkoA}{3QCCQEV6QyOHBfIXykwJEA}{}{}{m}, optionalJoin=Optional.empty}
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid dxsFw3OlT9izvLFXvAGBAw than local cluster uuid KjQI8jWoRUusEg9NEhXtFg, rejecting

Master2:
[2020-04-22T15:06:24,255][INFO][main][o.e.d.DiscoveryModule] [Master2] using discovery type [zen] and seed hosts providers [settings]
[2020-04-22T15:06:25,270][INFO][main][o.e.n.Node] [Master2] starting ...
[2020-04-22T15:06:26,979][INFO][elasticsearch[Master2][generic][T#4]][o.e.c.c.Coordinator] [Master2] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-Master1,7mv14SBoSvmNkgmi2FYQFQ,S4WT_EGjQri1PFaPpundsg}
[2020-04-22T15:06:27,345][INFO][elasticsearch[Master2][generic][T#4]][o.e.c.c.CoordinationState] [Master2] cluster UUID set to [dxsFw3OlT9izvLFXvAGBAw]
[2020-04-22T15:06:27,452][INFO][elasticsearch[Master2][clusterApplierService#updateTask][T#1]][o.e.c.s.ClusterApplierService] 
[Master2] master node changed {previous [], current [{Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{ip}{ip:port}{m}]}, 
added {{Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m}}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m}}

Master3:
[2020-04-22T15:06:20,175][INFO][main][o.e.n.Node] [Master3] starting ...
[2020-04-22T15:06:27,031][INFO][elasticsearch[EsMaster@10.162.146.158][masterService#updateTask][T#1]][o.e.c.s.MasterService] [Master3] elected-as-master ([2] nodes joined)[Master2}{S4WT_EGjQri1PFaPpundsg}{lRib4l65SuCWTCwxqNampg}{}{}{m} elect leader,{Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{Master3}{7mv14SBoSvmNkgmi2FYQFQ}{aAnNhNbXTje6qJKSgSZWsg}{}{}{m}]},added {{Master2}{S4WT_EGjQri1PFaPpundsg}{lRib4l65SuCWTCwxqNampg}{}{}{m}}

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 23, 2020, 8:49am UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/6 "2020-04-23T08:49:19Z")

</div>

Are you sure that all of these nodes have storage that persists across restarts?

---

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [April 23, 2020, 9:06am UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/7 "2020-04-23T09:06:25Z")

</div>

@DavidTurner I am sure that all of these nodes have storage that persists across restarts.  
For each master and node, I have mounted a disk in a different path.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 23, 2020, 9:11am UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/8 "2020-04-23T09:11:28Z")

</div>

In which case you have definitely configured `cluster.initial_master_nodes` differently at some point in their pasts. It's easily done, and the fix is described in the link I shared above.

---

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [April 23, 2020, 9:17am UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/9 "2020-04-23T09:17:19Z")

</div>

Unfortunately，i haven't modified it since I configured cluster.initial\_master\_nodes and discovery.seed\_hosts, and this is the first time I started the newly installed cluster.  
This is the second time I have encountered this problem.

I once asked such a question：

> [@Why node uuid replace by {bootstrap-placeholder}-{NodeName}](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869/2):
>
> At the time the cluster bootstrapped, it did not know the ID of master 3 so it uses a placeholder that will be replaced with master 3's ID later. This is quite a low-level implementation detail that almost certainly isn't relevant to your problem You did not set cluster.initial\_master\_nodes to the same value on all three nodes, or you have changed it after the first time you started these nodes. Not obviously, but if you changed cluster.initial\_master\_nodes then that was wrong.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 23, 2020, 9:56am UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/10 "2020-04-23T09:56:07Z")

</div>

> [@yc1024](#):
>
> this is the first time I started the newly installed cluster

That is not consistent with the logs you shared. These lines indicate that this is not the first time these nodes were started:

```auto
[2020-04-22T15:06:26,373][INFO][main][o.e.c.c.Coordinator] [Node1] cluster UUID [KjQI8jWoRUusEg9NEhXtFg]
[2020-04-22T15:06:26,419][INFO][main][o.e.c.c.Coordinator] [Master1] cluster UUID [KjQI8jWoRUusEg9NEhXtFg]

```

Specifically, the first time a node starts it logs a message saying `cluster UUID set to [...]` but the vital words `set to` are missing in these logs.

---

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [April 23, 2020, 3:50pm UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/11 "2020-04-23T15:50:01Z")

</div>

When the node is started for the first time, the log in `cluster UUID set to [...] `format must be printed?  
When will the log like `cluster UUID [...]` be printed?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 23, 2020, 5:28pm UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/12 "2020-04-23T17:28:49Z")

</div>

> [@yc1024](#):
>
> When will the log like `cluster UUID [...]` be printed?

That's logged when the node restarts having already joined a cluster.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2020, 5:32pm UTC](https://discuss.elastic.co/t/the-confusion-of-configuring-cluster-initial-master-nodes/229213/13 "2020-05-21T17:32:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.