One of master pod is not getting restarted

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I have deployed ECK 1.3.0 in GCP with 2 master and 3 data-ingest nodes. For backup purpose, referenced secret in secure settings in elasticsearch file and re applied. All the data-ingest and one of master pods are restarted. Only one master pod is not getting restarted. I have to made to restart manually . Could anyone suggest me how to get rid of this problem

2

The operator does not restart a Pod as long as some conditions are not met. Having the operator logs would help to understand why it was the case in your situation.

3

Below were the operator pod logs

{"log.level":"info","@timestamp":"2021-02-02T15:22:16.649Z","log.logger":"association.kb-es-association-controller","message":"Ending reconciliation run","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","iteration":1,"namespace":"elk-svc-510","kb_name":"kibana-config","took":0.170393058}
{"log.level":"info","@timestamp":"2021-02-02T15:22:16.821Z","log.logger":"observer","message":"Creating observer for cluster","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","namespace":"elk-svc-510","es_name":"elasticsearch-config"}
{"log.level":"info","@timestamp":"2021-02-02T15:22:25.950Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elk-svc-510","name":"elastic-licensing"}
I0202 15:22:27.000454       1 request.go:621] Throttling request took 1.043926284s, request: GET:https://172.24.0.1:443/apis/pxc.percona.com/v1-6-0?timeout=1m0s
I0202 15:22:37.050354       1 request.go:621] Throttling request took 5.94614821s, request: GET:https://172.24.0.1:443/apis/servicenetworking.cnrm.cloud.google.com/v1beta1?timeout=1m0s
{"log.level":"info","@timestamp":"2021-02-02T15:22:47.029Z","log.logger":"driver","message":"Could not update cluster license","service.version":"1.3.0+6db1914b","service.type":"eck","ecs.version":"1.4.0","err":"while getting current license level 503 Service Unavailable: ","namespace":"elk-svc-510","es_name":"elasticsearch-config"}
4

The operator logs failed predicates, some logs seem to be missing.

Looks like the cluster was not available, it does prevent the Pods from being restarted. I would have a look at the Elasticsearch cluster logs to understand where this 503 comes from.