# 3 node deployment with eck-operator logging cluster.initial\_master\_nodes is set WARN logs

**URL:** <https://discuss.elastic.co/t/3-node-deployment-with-eck-operator-logging-cluster-initial-master-nodes-is-set-warn-logs/354258>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [February 27, 2024, 4:57pm UTC](https://discuss.elastic.co/t/3-node-deployment-with-eck-operator-logging-cluster-initial-master-nodes-is-set-warn-logs/354258 "2024-02-27T16:57:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![coezdemir](https://avatars.discourse-cdn.com/v4/letter/c/7c8e57/32.png) [@coezdemir](https://discuss.elastic.co/u/coezdemir)\
**Post date:** [February 27, 2024, 4:57pm UTC](https://discuss.elastic.co/t/3-node-deployment-with-eck-operator-logging-cluster-initial-master-nodes-is-set-warn-logs/354258/1 "2024-02-27T16:57:50Z")

</div>

Hello,  
This will be my first post on here, I hope I get it right. Since I work for a Bank I cannot publish config files but I will try to give as much detail as I can.

We have deployed elasticsearch clusters using eck-operator with using ArgoCD applicationset (Helm version 2.11.0).  
We used default helm values for the operator.  
We apply Elasticsearch Kind CRDs as yamls to the namespaces with ArgoCD syncs.

Our initial cluster deployments are always working, no problems at all.  
We also have one of our teams testing a search app on another namespace, which has no problem with it.

Our NodeSet has 3 nodes with all roles possible, there are no master only nodes.  
We have a ceph-block storage class, which we used for elasticsearch-data pv claims.

On our clusters we see these logs sometimes constantly, sometimes once and dissapearing:

```auto
{"@timestamp":"2024-02-26T16:07:59.599Z", "log.level": "INFO", "message":"this node is locked into cluster UUID <UUID> and will not attempt further cluster bootstrapping", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"<node-name>","elasticsearch.cluster.name":"<cluster-name>"}
{"@timestamp":"2024-02-26T16:09:21.686Z", "log.level": "INFO", "message":"this node is locked into cluster UUID <UUID> and will not attempt further cluster bootstrapping", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"<node-name>","elasticsearch.cluster.name":"<cluster-name>"}

```

We previously built these clusters with helm charts, and the exact same thing happend as well. I know for a fact that after cluster deployment, elasticsearch containers should ignore the configuration for initial\_masters.

What would be the cause for this?

Regards  
Can

---

<div class="post-metadata">

**Author:** ![coezdemir](https://avatars.discourse-cdn.com/v4/letter/c/7c8e57/32.png) [@coezdemir](https://discuss.elastic.co/u/coezdemir)\
**Post date:** [February 28, 2024, 9:26am UTC](https://discuss.elastic.co/t/3-node-deployment-with-eck-operator-logging-cluster-initial-master-nodes-is-set-warn-logs/354258/2 "2024-02-28T09:26:25Z")

</div>

I would like to give an update, I have been going into pods and describing hte statefulsets we have both on prod and test clusters.

The config file is generated and kept inside the same namespace with pods, it secret name is --es-config and inside there I can actually see the discovery settings taking in a file which is called unicast-hosts and it is another configmap on the same namespace.

But there is absolutely nothing related to cluster.initial\_master\_nodes configuration on any elasticsearch.yml file I can find.

```auto
cluster:
    name: <cluster-name>
    routing:
        allocation:
            awareness:
                attributes: <k8s-node-attribute>
discovery:
    seed_hosts: []
    seed_providers: file -> unicast file with 3 pod IP:PORT

```

These are the only relevant lines I can find, and the elasticsearch.yml is being copied into /mnt/elastic-internal path and gets copied to the pods on restarts by prepare-fs.sh as far as I can see;

```auto
echo "Linking /mnt/elastic-internal/elasticsearch-config/elasticsearch.yml to /mnt/elastic-internal/elasticsearch-config-local/elasticsearch.yml"
ln -sf /mnt/elastic-internal/elasticsearch-config/elasticsearch.yml /mnt/elastic-internal/elasticsearch-config-local/elasticsearch.yml

```

When I bash into the running pods which throw these warning for nodes being locked for a cluster, I see the exact same elasticsearch.yml I see on k8s, which doesn't have anything related to cluster.initial\_master\_node setting.

Could there be another docker image default file somewhere that I am missing? or maybe this docker image over-writes some configs on restarts?

Can anybody help with this? I am trying to remove the cluster.initial\_master\_nodes setting but I cannot even find it.

One more edit: We have 4 clusters 2 for test 2 for production, which has absolutely the same k8s component configurations and elasticsearch yaml configurations. On test cluster, we only see this as a INFO once, but on production clusters "this node locked to cluster" message gets logged every 12 hours constantly.

Regards.

---

<div class="post-metadata">

**Author:** ![Thibault\_Richard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thibault_richard/32/50513_2.png) [@Thibault\_Richard](https://discuss.elastic.co/u/Thibault_Richard)\
**Post date:** [February 29, 2024, 5:28pm UTC](https://discuss.elastic.co/t/3-node-deployment-with-eck-operator-logging-cluster-initial-master-nodes-is-set-warn-logs/354258/3 "2024-02-29T17:28:19Z")

</div>

Hey,

Perhaps your cluster started with a volume already in use by another previous cluster, which might look like it's related to a bug in your volume provisioner that isn't properly recycling volumes?

See: [This node is locked into cluster UUID and will not attempt further cluster bootstrapping - Common causes and quick fixes](https://opster.com/analysis/elasticsearch-this-node-is-locked-into-cluster-uuid-and-will-not-attempt-further-cluster-bootstrapping/).

---

<div class="post-metadata">

**Author:** ![coezdemir](https://avatars.discourse-cdn.com/v4/letter/c/7c8e57/32.png) [@coezdemir](https://discuss.elastic.co/u/coezdemir)\
**Post date:** [March 1, 2024, 7:44am UTC](https://discuss.elastic.co/t/3-node-deployment-with-eck-operator-logging-cluster-initial-master-nodes-is-set-warn-logs/354258/4 "2024-03-01T07:44:04Z")

</div>

Thank you for the feedback,

We are using ceph-block volume claims, I will add the NodeSet configuration here about pvs:

```auto
volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: ceph-block 

```

Wouldn't this cause the node, to fail to join cluster? Even with warnings, we can see the nodes in \_cat/nodes and cluster is healthy. We have also tried indexing,searching,deleting etc. all seems to be working.

---

<div class="post-metadata">

**Author:** ![coezdemir](https://avatars.discourse-cdn.com/v4/letter/c/7c8e57/32.png) [@coezdemir](https://discuss.elastic.co/u/coezdemir)\
**Post date:** [March 16, 2024, 9:31am UTC](https://discuss.elastic.co/t/3-node-deployment-with-eck-operator-logging-cluster-initial-master-nodes-is-set-warn-logs/354258/5 "2024-03-16T09:31:27Z")

</div>

Hello,

We are still experiencing this, does anyone succesfully got rid of these warnings?

Regards.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2024, 9:32am UTC](https://discuss.elastic.co/t/3-node-deployment-with-eck-operator-logging-cluster-initial-master-nodes-is-set-warn-logs/354258/6 "2024-04-13T09:32:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.