# Single master node cluster dies when master node dies

**URL:** <https://discuss.elastic.co/t/single-master-node-cluster-dies-when-master-node-dies/281559>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [August 16, 2021, 2:04pm UTC](https://discuss.elastic.co/t/single-master-node-cluster-dies-when-master-node-dies/281559 "2021-08-16T14:04:59Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![iojas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iojas/32/117987_2.png) [@iojas](https://discuss.elastic.co/u/iojas)\
**Post date:** [August 16, 2021, 2:04pm UTC](https://discuss.elastic.co/t/single-master-node-cluster-dies-when-master-node-dies/281559/1 "2021-08-16T14:04:59Z")

</div>

I am using ECK to deploy an ES cluster. my setup has 1 master node and 3 data nodes. If for some reason master nodes dies it comes back up (thanks to statefulsets in kubernetes), but associates itself with a different cluster-ID. Thus it rejects the request of other data nodes who tries to join it.

but when I run an upgrade or something where kubernetes "safely" remove and bring back the master node cluster becomes healthy in a while.

I tried with 3 master nodes as well. but when I kill one node it is never able to join the existing cluster and the cluster goes to yellow state forever.

now my question is

what if some issue happens and the master node is not restarted safely how can I make sure my cluster formation is happening correctly.

Heres the yaml code for deploying ECK on my kubernetes cluster.

```auto
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: sifter-elastic-data-factory
spec:
  version: 7.14.0
  nodeSets:
    - name: master
      count: 1
      config:
        node.master: true
        node.data: false
        node.ingest: false
      podTemplate:
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 8Gi
                  cpu: 3000m
                limits:
                  memory: 8Gi
                  cpu: 3000m
              env:
                - name: ES_JAVA_OPTS
                  value: -Xms6g -Xmx6g
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 50Gi
            storageClassName: ssd
    - name: data
      count: 3
      config:
        node.master: false
        node.data: true
        node.ingest: true
      podTemplate:
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 8Gi
                  cpu: 3000m
                limits:
                  memory: 8Gi
                  cpu: 3000m
              env:
                - name: ES_JAVA_OPTS
                  value: -Xms6g -Xmx6g
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 60Gi
            storageClassName: ssd
  http:
    service:
      spec:
        type: ClusterIP
    tls:
      selfSignedCertificate:
        disabled: true

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 16, 2021, 10:00pm UTC](https://discuss.elastic.co/t/single-master-node-cluster-dies-when-master-node-dies/281559/2 "2021-08-16T22:00:37Z")

</div>

Are you using persistent volumes for the master?

---

<div class="post-metadata">

**Author:** ![iojas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iojas/32/117987_2.png) [@iojas](https://discuss.elastic.co/u/iojas)\
**Post date:** [August 17, 2021, 12:28pm UTC](https://discuss.elastic.co/t/single-master-node-cluster-dies-when-master-node-dies/281559/3 "2021-08-17T12:28:59Z")

</div>

Yes. persistent volume for master as well as for data. could this be an issue? @warkolm

---

<div class="post-metadata">

**Author:** ![iojas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iojas/32/117987_2.png) [@iojas](https://discuss.elastic.co/u/iojas)\
**Post date:** [August 17, 2021, 1:50pm UTC](https://discuss.elastic.co/t/single-master-node-cluster-dies-when-master-node-dies/281559/4 "2021-08-17T13:50:17Z")

</div>

@warkolm I guess this was the issue. when I removed the persistent storage on the master node, the cluster always comes back up without losing data.

followup question, what is the downside of not using persistent storage with the master node?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 17, 2021, 11:05pm UTC](https://discuss.elastic.co/t/single-master-node-cluster-dies-when-master-node-dies/281559/5 "2021-08-17T23:05:41Z")

</div>

Your masters need persistent storage - [Node | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/modules-node.html#master-node)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2021, 11:05pm UTC](https://discuss.elastic.co/t/single-master-node-cluster-dies-when-master-node-dies/281559/6 "2021-09-14T23:05:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.