# 503 Service Unavailable - Master not discovered error in AKS

**URL:** <https://discuss.elastic.co/t/503-service-unavailable-master-not-discovered-error-in-aks/273713>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [May 22, 2021, 1:34am UTC](https://discuss.elastic.co/t/503-service-unavailable-master-not-discovered-error-in-aks/273713 "2021-05-22T01:34:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![pindropviolence](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pindropviolence/32/101494_2.png) [@pindropviolence](https://discuss.elastic.co/u/pindropviolence)\
**Post date:** [May 22, 2021, 1:34am UTC](https://discuss.elastic.co/t/503-service-unavailable-master-not-discovered-error-in-aks/273713/1 "2021-05-22T01:34:55Z")

</div>

HI,  
I have a AKS cluster (k1) with 3 nodes. I have deployed ECK operator using all\_in\_one.yaml.  
I have another AKS cluster(k2) which is running my application stack , i have installed filebeats in this cluster (k2) and it is shipping the logs to elasticsearch to another k8s cluster (k1).  
My k2 cluster configuration is 3 nodes (4 vCPU, 16 GiB RAM , 32GiB Temp Storage).  
My elasticsearch.yaml:

```auto
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: dev-elasticsearch
spec:
  version: 7.9.2
  http:
    # Disable https
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        type: LoadBalancer
  nodeSets:
  - name: default
    count: 3
    config:
      node.master: true
      node.data: true
      node.store.allow_mmap: false
      xpack.security.enabled: true
      discovery.seed_hosts:
         - 10.x.x.x
         - 10.x.x.x
         - 10.x.x.x

```

When i start the start es and kibana , everything works fine for about 30 mins and then i have service unavailable error in es  
`{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}`

Another thing i noticed is elastic-internal-init-filesystem is in Terminated state.

1. Can someone help me resolve this issue ?
2. Also how can i write the logs to a az storage with more space instead on default PV ?

Thanks for your help !!

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [May 25, 2021, 8:09am UTC](https://discuss.elastic.co/t/503-service-unavailable-master-not-discovered-error-in-aks/273713/2 "2021-05-25T08:09:57Z")

</div>

You should not set discovery.seed\_hosts yourself. The ECK operator takes care of it for you, and maintains it up-to-date at all time.  
Can you try removing it from the configuration? It will likely help with node discovery.

---

<div class="post-metadata">

**Author:** ![pindropviolence](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pindropviolence/32/101494_2.png) [@pindropviolence](https://discuss.elastic.co/u/pindropviolence)\
**Post date:** [May 28, 2021, 4:01pm UTC](https://discuss.elastic.co/t/503-service-unavailable-master-not-discovered-error-in-aks/273713/3 "2021-05-28T16:01:43Z")

</div>

Hi @sebgl - Thanks , i will remove discovery.seed\_hosts, however i am not seeing any issues now having that in the elasticsearch manifest. The workloads are running fine now in order to resolve my issue I had to put more space into the pods , there is around 20 GiB of data/per from my application kubernetes cluster and when i used the default (2 GiB) the space ran out almost in 20-30 mins and rendered Service unavailable.

```auto
 volumeClaimTemplates:
    - metadata: 
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 90Gi
        storageClassName: azurefile
        persistentVolumeReclaimPolicy: Delete

```

Big Thanks to this community - i have been able to make headway reading articles and issue resolution.  
Onto my other question - how can i have rolling peristant volumes ? I mean i am using around 100 GiB of persistant volume claims but it will run out someday probably in a month so I want to have a mechanism that will archive my pod volume data and make that always available. Please let me know if there is a way to do that ?

---

<div class="post-metadata">

**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\
**Post date:** [May 31, 2021, 7:18am UTC](https://discuss.elastic.co/t/503-service-unavailable-master-not-discovered-error-in-aks/273713/4 "2021-05-31T07:18:03Z")

</div>

> it will run out someday probably in a month so I want to have a mechanism that will archive my pod volume data and make that always available

Sounds like you may be interested in using the cold and frozen tiers. The idea is basically that you can roll data after some time (or shard size), and back it up in a snapshot where it still says searchable.  
See [Data tiers | Elasticsearch Guide [master] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/data-tiers.html) and [Directly search S3 with the new frozen tier | Elastic Blog](https://www.elastic.co/blog/introducing-elasticsearch-frozen-tier-searchbox-on-s3).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2021, 7:18am UTC](https://discuss.elastic.co/t/503-service-unavailable-master-not-discovered-error-in-aks/273713/5 "2021-06-28T07:18:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.