# Elasticsearch cluster pods are not restarting for version upgrade

**URL:** <https://discuss.elastic.co/t/elasticsearch-cluster-pods-are-not-restarting-for-version-upgrade/248090>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [September 10, 2020, 1:08am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-pods-are-not-restarting-for-version-upgrade/248090 "2020-09-10T01:08:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\
**Post date:** [September 10, 2020, 1:08am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-pods-are-not-restarting-for-version-upgrade/248090/1 "2020-09-10T01:08:14Z")

</div>

- Upgrading from ES `7.6.1 to 7.9.1`
- Upgrading elastic-cloud-operator from `1.1.2 to 1.2.1`
- cluster status is green with no issues  
For some reason our operator is stuck at "do\_not\_restart\_healthy\_node\_if\_MaxUnavailable\_reached" message during the upgrade:

```auto
{"log.level":"info","@timestamp":"2020-09-10T01:04:04.001Z","log.logger":"zen2","message":"Ensuring no voting exclusions are set","service.version":"1.2.1-b5316231","service.type":"eck","ecs.version":"1.4.0","namespace":"elastic-system","es_name":"elasticsearch"}
{"log.level":"info","@timestamp":"2020-09-10T01:04:04.016Z","log.logger":"migrate-data","message":"Setting routing allocation excludes","service.version":"1.2.1-b5316231","service.type":"eck","ecs.version":"1.4.0","namespace":"elastic-system","es_name":"elasticsearch","value":"none_excluded"}
{"log.level":"info","@timestamp":"2020-09-10T01:04:04.032Z","log.logger":"driver","message":"Cannot restart some nodes for upgrade at this time","service.version":"1.2.1-b5316231","service.type":"eck","ecs.version":"1.4.0","namespace":"elastic-system","es_name":"elasticsearch","failed_predicates":{"do_not_restart_healthy_node_if_MaxUnavailable_reached":["elasticsearch-es-data-5","elasticsearch-es-data-4","elasticsearch-es-data-3","elasticsearch-es-data-2","elasticsearch-es-masters-2","elasticsearch-es-masters-1","elasticsearch-es-masters-0"]}}

```

our elasticsearch config (SHORT VERSION):

```auto
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  annotations:
    common.k8s.elastic.co/controller-version: 1.2.1
    meta.helm.sh/release-name: elasticsearch
    meta.helm.sh/release-namespace: elastic-system
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 7.9.1
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
    elasticsearch.k8s.elastic.co/statefulset-name: elasticsearch-es-data
  name: elasticsearch
  namespace: elastic-system
spec:
 nodeSets:
  - config:
    count: 3
    name: masters
  - config:
    count: 6
    name: data
  updateStrategy:
    changeBudget:
      maxSurge: 1
      maxUnavailable: 0
  version: 7.9.1
status:
  availableNodes: 9
  health: green
  phase: ApplyingChanges

```

elastic-cloud-operator statefulset (SHORT VERSION):

```auto
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    meta.helm.sh/release-name: elastic-cloud-operator
    meta.helm.sh/release-namespace: elastic-system
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 1.2.1
    control-plane: elastic-cloud-operator
  name: elastic-cloud-operator
  namespace: elastic-system
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      control-plane: elastic-cloud-operator
  serviceName: elastic-cloud-operator
  template:
    spec:
      containers:
      - args:
        - manager
        - --enable-webhook
        - --log-verbosity=0
        - --metrics-port=9090
        env:
        - name: OPERATOR_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: OPERATOR_IMAGE
          value: docker.elastic.co/eck/eck-operator:1.2.1
        - name: WEBHOOK_SECRET
          value: elastic-cloud-operator-webhook-server-cert
        image: docker.elastic.co/eck/eck-operator:1.2.1
        imagePullPolicy: IfNotPresent
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
status:
  collisionCount: 0
  currentReplicas: 1
  currentRevision: elastic-cloud-operator-75ccb4b8bb
  observedGeneration: 5
  readyReplicas: 1
  replicas: 1
  updateRevision: elastic-cloud-operator-75ccb4b8bb
  updatedReplicas: 1

```

---

<div class="post-metadata">

**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\
**Post date:** [September 10, 2020, 8:20am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-pods-are-not-restarting-for-version-upgrade/248090/2 "2020-09-10T08:20:41Z")

</div>

I think the problem here is that you are running with `maxUnavailable: 0` that leaves no room for the operator to remove a node in order to upgrade it. For a rolling upgrade at least one Pod at a time needs to be taken down for upgrade so you have to set it to at least `1` for the upgrade to go through. See also [https://www.elastic.co/guide/en/cloud-on-k8s/1.2/k8s-update-strategy.html](https://www.elastic.co/guide/en/cloud-on-k8s/1.2/k8s-update-strategy.html)

---

<div class="post-metadata">

**Author:** ![jbonzo](https://avatars.discourse-cdn.com/v4/letter/j/8e7dd6/32.png) [@jbonzo](https://discuss.elastic.co/u/jbonzo)\
**Post date:** [October 22, 2020, 3:14am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-pods-are-not-restarting-for-version-upgrade/248090/3 "2020-10-22T03:14:24Z")

</div>

I am running on a similar setup, but I am using the default update strategy that has `maxUnavailable: 1`.

- ES version is at `7.8.0` and not changing
- Upgrading elastic-cloud operator from `1.1.0` to `1.2.1`

Besides those differences all my symptoms are the same.

edit:  
Added the logs for the reconciliation cycle.

```auto
[
  {
    "log.level": "debug",
    "@timestamp": "2020-10-22T03:20:26.264Z",
    "log.logger": "observer",
    "message": "Retrieving cluster state",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "es_name": "elky",
    "namespace": "elastic-cluster"
  },
  {
    "log.level": "info",
    "@timestamp": "2020-10-22T03:20:28.749Z",
    "log.logger": "elasticsearch-controller",
    "message": "Starting reconciliation run",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "iteration": 147,
    "namespace": "elastic-cluster",
    "es_name": "elky"
  },
  {
    "log.level": "debug",
    "@timestamp": "2020-10-22T03:20:28.749Z",
    "log.logger": "es-validation",
    "message": "validate create",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "name": "elky"
  },
  {
    "log.level": "info",
    "@timestamp": "2020-10-22T03:20:29.638Z",
    "log.logger": "zen2",
    "message": "Ensuring no voting exclusions are set",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "namespace": "elastic-cluster",
    "es_name": "elky"
  },
  {
    "log.level": "info",
    "@timestamp": "2020-10-22T03:20:29.790Z",
    "log.logger": "migrate-data",
    "message": "Setting routing allocation excludes",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "namespace": "elastic-cluster",
    "es_name": "elky",
    "value": "none_excluded"
  },
  {
    "log.level": "debug",
    "@timestamp": "2020-10-22T03:20:29.990Z",
    "log.logger": "driver",
    "message": "Applying predicates",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "maxUnavailableReached": false,
    "allowedDeletions": 1
  },
  {
    "log.level": "info",
    "@timestamp": "2020-10-22T03:20:29.993Z",
    "log.logger": "driver",
    "message": "Cannot restart some nodes for upgrade at this time",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "namespace": "elastic-cluster",
    "es_name": "elky",
    "failed_predicates": {
      "only_restart_healthy_node_if_green_or_yellow": [
        ...
        "elky-es-logging-2",
        "elky-es-logging-1",
        "elky-es-logging-0",
        "elky-es-master-2",
        "elky-es-master-1",
        "elky-es-master-0"
      ]
    }
  },
  {
    "log.level": "debug",
    "@timestamp": "2020-10-22T03:20:29.993Z",
    "log.logger": "driver",
    "message": "No pod deleted during rolling upgrade",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "es_name": "elky",
    "namespace": "elastic-cluster"
  },
  {
    "log.level": "debug",
    "@timestamp": "2020-10-22T03:20:30.020Z",
    "log.logger": "driver",
    "message": "Statefulset not reconciled",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "reason": "pod not upgraded"
  },
  {
    "log.level": "info",
    "@timestamp": "2020-10-22T03:20:30.021Z",
    "log.logger": "elasticsearch-controller",
    "message": "Ending reconciliation run",
    "service.version": "1.2.1-b5316231",
    "service.type": "eck",
    "ecs.version": "1.4.0",
    "iteration": 147,
    "namespace": "elastic-cluster",
    "es_name": "elky",
    "took": 1.271987487
  }
]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 8:12am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-pods-are-not-restarting-for-version-upgrade/248090/4 "2022-11-04T08:12:50Z")

</div>