# How to deploy elastic on k8s to avoid join validation on cluster state with a different cluster uuid error

**URL:** <https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850>\
**Category:** Elasticsearch\
**Created:** [July 17, 2019, 2:25am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850 "2019-07-17T02:25:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![yuecong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuecong/32/49061_2.png) [@yuecong](https://discuss.elastic.co/u/yuecong)\
**Post date:** [July 17, 2019, 2:25am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/1 "2019-07-17T02:25:30Z")

</div>

I am trying to deploy elastic stack on top of k8s with 2 client nodes, 3 master nodes and 3 data nodes. And all of them are configured on a different node now.

For the 2 client nodes. I am using k8s deployment, and for master nodes and data nodes I am using k8s statefulset. it worked for the first time if all nodes are fresh, but when I updated the master nodes's statedulset, I got the error as

> Blockquote  
> **Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid gR1WEwUpRXynUdhOyF2axA than local cluster uuid 1gwMWNx0TPSd3j9-hxlgcA, rejecting**

Here is the related k8s deployment for each node

A common setting for all the 3 type of nodes, but changed the setting by controlling the env.

> apiVersion: v1  
> kind: ConfigMap  
> metadata:  
> name: elasticsearch  
> namespace: elasticsearch  
> labels:  
> app: elasticsearch
> 
> data:  
> elasticsearch.yml: |-  
> cluster:  
> name: ${CLUSTER\_NAME}  
> initial\_master\_nodes: "es-master-0,es-master-1,es-master-2"

> ```
> node:
> master: ${NODE_MASTER}
> data: ${NODE_DATA}
> name: ${NODE_NAME}
> ingest: ${NODE_INGEST}
> max_local_storage_nodes: 1
> attr.box_type: hot
> processors: ${PROCESSORS:1}
> network.host: ${NETWORK_HOST}
> path:
> data: /usr/share/elasticsearch/data
> logs: /usr/share/elasticsearch/logs
> http:
> compression: true
> discovery:
> seed_hosts: ${DISCOVERY_SERVICE}
> 
> ```

master node:  
a headless service for master nodes

> apiVersion: v1  
> kind: Service  
> metadata:  
> name: elasticsearch-discovery  
> namespace: elasticsearch  
> labels:  
> component: elasticsearch  
> role: master  
> spec:  
> selector:  
> component: elasticsearch  
> role: master  
> ports:
> 
> - name: transport  
> port: 9300  
> protocol: TCP  
> clusterIP: None

configs for master node

> ```
> - name: elasticsearch
> env:
> - name: CLUSTER_NAME
> value: logs001
> - name: NUMBER_OF_MASTERS
> value: "3"
> - name: NODE_MASTER
> value: "true"
> - name: NODE_INGEST
> value: "false"
> - name: NODE_DATA
> value: "false"
> - name: NETWORK_HOST
> value: "0.0.0.0"
> - name: NODE_NAME
> valueFrom:
> fieldRef:
> fieldPath: metadata.name
> - name: DISCOVERY_SERVICE
> value: elasticsearch-discovery
> - name: KUBERNETES_NAMESPACE
> valueFrom:
> fieldRef:
> fieldPath: metadata.namespace
> - name: PROCESSORS
> valueFrom:
> resourceFieldRef:
> resource: limits.cpu
> - name: ES_JAVA_OPTS
> value: -Xms48g -Xmx48g
> 
> ```

configs for data nodes

> ```
> env:
> - name: CLUSTER_NAME
> value: logs001
> - name: NODE_MASTER
> value: "false"
> - name: NODE_INGEST
> value: "false"
> - name: NETWORK_HOST
> value: "_eth0_"
> - name: NUMBER_OF_MASTERS
> value: "3"
> - name: NODE_NAME
> valueFrom:
> fieldRef:
> fieldPath: metadata.name
> - name: DISCOVERY_SERVICE
> value: elasticsearch-discovery
> - name: KUBERNETES_NAMESPACE
> valueFrom:
> fieldRef:
> fieldPath: metadata.namespace
> - name: NODE_DATA
> value: "true"
> - name: PROCESSORS
> valueFrom:
> resourceFieldRef:
> resource: limits.cpu
> - name: ES_JAVA_OPTS
> value: -Xms48g -Xmx48g
> 
> ```

configs for client node

> ```
> env:
> - name: CLUSTER_NAME
> value: logs001
> - name: NUMBER_OF_MASTERS
> value: "3"
> - name: NODE_MASTER
> value: "false"
> - name: NODE_INGEST
> value: "true"
> - name: NODE_DATA
> value: "false"
> - name: NETWORK_HOST
> value: "_eth0_"
> - name: NODE_NAME
> valueFrom:
> fieldRef:
> fieldPath: metadata.name
> - name: DISCOVERY_SERVICE
> value: elasticsearch-discovery
> - name: KUBERNETES_NAMESPACE
> valueFrom:
> fieldRef:
> fieldPath: metadata.namespace
> - name: PROCESSORS
> valueFrom:
> resourceFieldRef:
> resource: limits.cpu
> - name: ES_JAVA_OPTS
> value: -Xms6g -Xmx6g
> 
> ```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 17, 2019, 5:18am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/2 "2019-07-17T05:18:55Z")

</div>

I think this means that your master nodes all restarted at once and were not using persistent storage, so they lost the cluster metadata. Your master nodes must use storage that persists across restarts.

---

<div class="post-metadata">

**Author:** ![yuecong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuecong/32/49061_2.png) [@yuecong](https://discuss.elastic.co/u/yuecong)\
**Post date:** [July 17, 2019, 5:22am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/3 "2019-07-17T05:22:34Z")

</div>

Thanks for the reply.  
master node is using k8s statefulset, so it only have one pod restarted at one time.  
ButI am not using persistent storage for master node. Let me check whether the issue will get resolved if I adds persistent disks for master nodes. btw, I think master node only has meta data, so is it ok a master node only use small amount of disks?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 17, 2019, 5:28am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/4 "2019-07-17T05:28:37Z")

</div>

> [@yuecong](#):
>
> it only have one pod restarted at one time.

I think this is not the case, or else the config you quote above is not the one that Elasticsearch is using.

> [@yuecong](#):
>
> is it ok a master node only use small amount of disks?

Yes, they normally need less storage than data nodes.

---

<div class="post-metadata">

**Author:** ![yuecong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuecong/32/49061_2.png) [@yuecong](https://discuss.elastic.co/u/yuecong)\
**Post date:** [July 17, 2019, 5:36am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/5 "2019-07-17T05:36:19Z")

</div>

Appreciate your insights here

Here is the sts for the master node.

> apiVersion: apps/v1  
> kind: StatefulSet  
> metadata:  
> labels:  
> component: elasticsearch  
> role: master  
> name: es-master  
> namespace: elasticsearch  
> spec:  
> serviceName: elasticsearch-master  
> replicas: 3 # Number of Elasticsearch master nodes to deploy  
> selector:  
> matchLabels:  
> component: elasticsearch  
> role: master  
> template:  
> metadata:  
> labels:  
> component: elasticsearch  
> role: master

Could you also suggest whether by using k8s's statefulset is sufficient? Or should I even wait for one master node is fully back to form a cluster?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 17, 2019, 6:54am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/6 "2019-07-17T06:54:17Z")

</div>

Sorry, I'm not the best person to help with K8s-specific questions. I believe it's possible to use a statefulset, yes, although you might prefer to use the [Elasticsearch operator](https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond).

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 17, 2019, 6:54am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/7 "2019-07-17T06:54:54Z")

</div>

> [@yuecong](#):
>
> should I even wait for one master node is fully back to form a cluster?

I don't really understand this question in the context of Kubernetes, but in general you should try to avoid restarting more than one node at once.

---

<div class="post-metadata">

**Author:** ![yuecong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuecong/32/49061_2.png) [@yuecong](https://discuss.elastic.co/u/yuecong)\
**Post date:** [July 17, 2019, 7:00am UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/8 "2019-07-17T07:00:16Z")

</div>

Thanks

---

<div class="post-metadata">

**Author:** ![yuecong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuecong/32/49061_2.png) [@yuecong](https://discuss.elastic.co/u/yuecong)\
**Post date:** [July 17, 2019, 2:38pm UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/9 "2019-07-17T14:38:09Z")

</div>

This issue is solved as @DavidTurner suggested like a charm! Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2019, 2:38pm UTC](https://discuss.elastic.co/t/how-to-deploy-elastic-on-k8s-to-avoid-join-validation-on-cluster-state-with-a-different-cluster-uuid-error/190850/10 "2019-08-14T14:38:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.