# Why node uuid replace by {bootstrap-placeholder}-{NodeName}

**URL:** <https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869>\
**Category:** Elasticsearch\
**Created:** [March 31, 2020, 1:20pm UTC](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869 "2020-03-31T13:20:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [March 31, 2020, 1:20pm UTC](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869/1 "2020-03-31T13:20:16Z")

</div>

ES version: 7.6.0

I have 3 master and 3 node in my cluster.

This is the first time i start the cluster,i have configed `discovery.seed_hosts` and `cluster.initial_master_nodes` as follow:

```auto
discovery.seed_hosts: ipOfMaster1：9300,ipOfMaster2：9300,ipOfMaster3：9300
cluster.initial_master_nodes:nameOfMaster1,nameOfMaster2,nameOfMaster3

```

```auto
curl -X GET "localhost:9200/_cluster/state?filter_path=metadata.cluster_coordination.last_committed_config&pretty"

{
"metadata" : {
    "cluster_coordination" : {
      "last_committed_config" : [
        "vFexMAR6T32k3qUC2ZBahg",     
        "ZMZgi3_qSX-soQLwkQZZfA",     
        "{bootstrap-placeholder}-nameOfMaster3" 
      ]
    }
  }
}

```

```auto
vFexMAR6T32k3qUC2ZBahg: the uuid of master1
ZMZgi3_qSX-soQLwkQZZfA: the uuid of master2
{bootstrap-placeholder}-nameOfMaster3 : {bootstrap-placeholder}-nameOfMaster3

```

I using `GET /` to checking the uuid of each node:

master1 and master2 have the same cluster\_uuid, but the master3's uuid is _na_.

I find the following Exception in master3's log:

```auto
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid d5OTAb8FTe-5iIZeHQYYVw than local cluster uuid oP_otesSTv2qItFqBHPPVw, rejecting

```

1. Why the uuid of master3 were replaced by {bootstrap-placeholder}-nameOfMaster3?
2. Why master3 have a different cluster uuid ? is it brain-split?
3. Is there something wrong with my configuration?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 31, 2020, 3:14pm UTC](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869/2 "2020-03-31T15:14:06Z")

</div>

> [@yc1024](#):
>
> 1. Why the uuid of master3 were replaced by {bootstrap-placeholder}-nameOfMaster3?

At the time the cluster bootstrapped, it did not know the ID of master 3 so it uses a placeholder that will be replaced with master 3's ID later. This is quite a low-level implementation detail that almost certainly isn't relevant to your problem

> [@yc1024](#):
>
> 1. Why master3 have a different cluster uuid ? is it brain-split?

You did not set `cluster.initial_master_nodes` to the same value on all three nodes, or you have changed it after the first time you started these nodes.

> [@yc1024](#):
>
> 1. Is there something wrong with my configuration?

Not obviously, but if you changed `cluster.initial_master_nodes` then that was wrong.

---

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [April 1, 2020, 2:15am UTC](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869/3 "2020-04-01T02:15:45Z")

</div>

@DavidTurner Thanks for your reply.

> [@DavidTurner](#):
>
> You did not set `cluster.initial_master_nodes` to the same value on all three nodes, or you have changed it after the first time you started these nodes.

Unfortunately，i had set `cluster.initial_master_nodes` to the same value on all three nodes， and never change them.

I found in the log of master3 that when master3 was started, an exception was thrown in a third-party plugin and the TCP connection was closed. Is this the root cause of this problem?

```auto
 exception caught on transport layer [Netty4TcpChannel] , closing connection

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 1, 2020, 6:28am UTC](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869/4 "2020-04-01T06:28:45Z")

</div>

No, I don't think a connection failure in a plugin could be related.

One other possibility is that your master nodes do not have storage that persists across restarts.

---

<div class="post-metadata">

**Author:** ![yc1024](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yc1024/32/42644_2.png) [@yc1024](https://discuss.elastic.co/u/yc1024)\
**Post date:** [April 1, 2020, 7:44am UTC](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869/5 "2020-04-01T07:44:07Z")

</div>

I saw the description of [Auto-bootstrapping in development mode](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/modules-discovery-bootstrap-cluster.html#_auto_bootstrapping_in_development_mode) :

If you start an Elasticsearch node without configuring these settings then it will start up in development mode and auto-bootstrap itself into a new cluster. If you start some Elasticsearch nodes on different hosts then by default they will not discover each other and will form a different cluster on each host. Elasticsearch will not merge separate clusters together after they have formed, even if you subsequently try and configure all the nodes into a single cluster. This is because there is no way to merge these separate clusters together without a risk of data loss. You can tell that you have formed separate clusters by checking the cluster UUID reported by `GET /` on each node. If you intended to form a single cluster then you should start again:

- Shut down all the nodes.
- Completely wipe each node by deleting the contents of their [data folders](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/modules-node.html#data-path).
- Configure `cluster.initial_master_nodes` as described above.
- Restart all the nodes and verify that they have formed a single cluster.

The master3 was mounted with a data disk. I deleted `nodes/0/_state` on master3 and restarted master3, then master3 joined the cluster.

Is master3 auto-bootstrapping in development mode?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 1, 2020, 7:58am UTC](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869/6 "2020-04-01T07:58:12Z")

</div>

> [@yc1024](#):
>
> Is master3 auto-bootstrapping in development mode?

No, auto-bootstrapping is disabled if `cluster.initial_master_nodes` is set, which you are saying has always been the case.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 29, 2020, 7:58am UTC](https://discuss.elastic.co/t/why-node-uuid-replace-by-bootstrap-placeholder-nodename/225869/7 "2020-04-29T07:58:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.