# One of master pod is not getting restarted

**URL:** <https://discuss.elastic.co/t/one-of-master-pod-is-not-getting-restarted/262985>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [February 2, 2021, 1:31pm UTC](https://discuss.elastic.co/t/one-of-master-pod-is-not-getting-restarted/262985 "2021-02-02T13:31:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![knagasri](https://avatars.discourse-cdn.com/v4/letter/k/47e85d/32.png) [@knagasri](https://discuss.elastic.co/u/knagasri)\
**Post date:** [February 2, 2021, 1:31pm UTC](https://discuss.elastic.co/t/one-of-master-pod-is-not-getting-restarted/262985/1 "2021-02-02T13:31:57Z")

</div>

I have deployed ECK 1.3.0 in GCP with 2 master and 3 data-ingest nodes. For backup purpose, referenced secret in secure settings in elasticsearch file and re applied. All the data-ingest and one of master pods are restarted. Only one master pod is not getting restarted. I have to made to restart manually . Could anyone suggest me how to get rid of this problem

---

<div class="post-metadata">

**Author:** ![michael.morello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael.morello/32/47448_2.png) [@michael.morello](https://discuss.elastic.co/u/michael.morello)\
**Post date:** [February 2, 2021, 1:56pm UTC](https://discuss.elastic.co/t/one-of-master-pod-is-not-getting-restarted/262985/2 "2021-02-02T13:56:59Z")

</div>

> [@knagasri](#):
>
> Only one master pod is not getting restarted.

The operator does not restart a Pod as long as some conditions are not met. Having the operator logs would help to understand why it was the case in your situation.

---

<div class="post-metadata">

**Author:** ![knagasri](https://avatars.discourse-cdn.com/v4/letter/k/47e85d/32.png) [@knagasri](https://discuss.elastic.co/u/knagasri)\
**Post date:** [February 2, 2021, 3:23pm UTC](https://discuss.elastic.co/t/one-of-master-pod-is-not-getting-restarted/262985/3 "2021-02-02T15:23:45Z")

</div>

Below were the operator pod logs

```auto
{"log.level":"info","@timestamp":"2021-02-02T15:22:16.649Z","log.logger":"association.kb-es-association-controller","message":"Ending reconciliation run","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","iteration":1,"namespace":"elk-svc-510","kb_name":"kibana-config","took":0.170393058}
{"log.level":"info","@timestamp":"2021-02-02T15:22:16.821Z","log.logger":"observer","message":"Creating observer for cluster","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","namespace":"elk-svc-510","es_name":"elasticsearch-config"}
{"log.level":"info","@timestamp":"2021-02-02T15:22:25.950Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elk-svc-510","name":"elastic-licensing"}
I0202 15:22:27.000454 1 request.go:621] Throttling request took 1.043926284s, request: GET:https://172.24.0.1:443/apis/pxc.percona.com/v1-6-0?timeout=1m0s
I0202 15:22:37.050354 1 request.go:621] Throttling request took 5.94614821s, request: GET:https://172.24.0.1:443/apis/servicenetworking.cnrm.cloud.google.com/v1beta1?timeout=1m0s
{"log.level":"info","@timestamp":"2021-02-02T15:22:47.029Z","log.logger":"driver","message":"Could not update cluster license","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","err":"while getting current license level 503 Service Unavailable: ","namespace":"elk-svc-510","es_name":"elasticsearch-config"}

```

---

<div class="post-metadata">

**Author:** ![michael.morello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael.morello/32/47448_2.png) [@michael.morello](https://discuss.elastic.co/u/michael.morello)\
**Post date:** [February 3, 2021, 9:04am UTC](https://discuss.elastic.co/t/one-of-master-pod-is-not-getting-restarted/262985/4 "2021-02-03T09:04:35Z")

</div>

[The operator logs failed predicates](https://github.com/elastic/cloud-on-k8s/blob/e52945b66f33debc67c0c1028714eec3216edfed/pkg/controller/elasticsearch/driver/upgrade_predicates.go#L106), some logs seem to be missing.

> [@knagasri](#):
>
> `503 Service Unavailable`

Looks like the cluster was not available, it does prevent the Pods from being restarted. I would have a look at the Elasticsearch cluster logs to understand where this 503 comes from.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 8:23am UTC](https://discuss.elastic.co/t/one-of-master-pod-is-not-getting-restarted/262985/5 "2022-11-04T08:23:09Z")

</div>