# ECK Operator stuck in ApplyingChanges state after upgrade to 1.4.0

**URL:** <https://discuss.elastic.co/t/eck-operator-stuck-in-applyingchanges-state-after-upgrade-to-1-4-0/265726>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [February 28, 2021, 2:37pm UTC](https://discuss.elastic.co/t/eck-operator-stuck-in-applyingchanges-state-after-upgrade-to-1-4-0/265726 "2021-02-28T14:37:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![barracuda-charles](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@barracuda-charles](https://discuss.elastic.co/u/barracuda-charles)\
**Post date:** [February 28, 2021, 2:37pm UTC](https://discuss.elastic.co/t/eck-operator-stuck-in-applyingchanges-state-after-upgrade-to-1-4-0/265726/1 "2021-02-28T14:37:03Z")

</div>

I recently upgraded to 1.4.0 and the operator is stuck in the ApplyingChanges state. I expanded a few volumes on data nodes which caused the cluster to enter a yellow state temporarily however the operator stayed green the whole time. It has been over 12 hours since the upgrade to 1.4.0 (and the cluster has been in a green state since then:

```auto
$ k get es
NAME HEALTH NODES VERSION PHASE AGE
search green 63 6.8.6 ApplyingChanges 289d

```

I don't see anything relevant in the logs for the operator:

```auto
{"log.level":"info","@timestamp":"2021-02-28T14:25:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:27:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:29:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:31:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:33:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:35:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:37:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}

```

It looks like the es cluster has an invalid annotation:

```auto
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  annotations:
    common.k8s.elastic.co/controller-version: 0.0.0-UNKNOWN

```

This was an upgrade from `1.2.x` to `1.4.0` -- should I try downgrading to `1.3.x` and then back to `1.4.0` to resolve the inconsistency, or perhaps edit the annotation to something suitable?

When I change the manifest I see the following in the log:

```auto
{"log.level":"info","@timestamp":"2021-02-28T20:00:51.333Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","iteration":3,"namespace":"default","es_name":"search"}
{"log.level":"info","@timestamp":"2021-02-28T20:00:51.333Z","log.logger":"annotation","message":"Resource was created with older version of operator, will not take action","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","controller_version":"1.4.0","resource_controller_version":"0.0.0-UNKNOWN","namespace":"default","name":"search"}
{"log.level":"info","@timestamp":"2021-02-28T20:00:51.333Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","iteration":3,"namespace":"default","es_name":"search","took":0.000139559}

```

---

<div class="post-metadata">

**Author:** ![barracuda-charles](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@barracuda-charles](https://discuss.elastic.co/u/barracuda-charles)\
**Post date:** [February 28, 2021, 8:27pm UTC](https://discuss.elastic.co/t/eck-operator-stuck-in-applyingchanges-state-after-upgrade-to-1-4-0/265726/2 "2021-02-28T20:27:44Z")

</div>

I ended up setting the `common.k8s.elastic.co/controller-version` annotation to `1.2.2` (the previous version I upgraded from) and it seems that the operator picked this up as "valid" and started applying requested changes.

---

<div class="post-metadata">

**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\
**Post date:** [March 1, 2021, 12:37pm UTC](https://discuss.elastic.co/t/eck-operator-stuck-in-applyingchanges-state-after-upgrade-to-1-4-0/265726/3 "2021-03-01T12:37:08Z")

</div>

The Elasticsearch cluster existed already in 1.2.2? Or did you create it only after upgrading to 1.4.0?  
Also: was there a point in time where both instances of the operator ran at the same time?

---

<div class="post-metadata">

**Author:** ![barracuda-charles](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@barracuda-charles](https://discuss.elastic.co/u/barracuda-charles)\
**Post date:** [March 1, 2021, 3:20pm UTC](https://discuss.elastic.co/t/eck-operator-stuck-in-applyingchanges-state-after-upgrade-to-1-4-0/265726/4 "2021-03-01T15:20:12Z")

</div>

The cluster existed already in 1.2.2

There should have only been 1 operator at a time, I did the following to upgrade from 1.2.2:

```auto
kubectl apply -f https://download.elastic.co/downloads/eck/1.4.0/all-in-one.yaml

```

Also, I have a yaml file that I apply to edit the cluster -- should I keep the annotation in that yaml file?

---

<div class="post-metadata">

**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\
**Post date:** [March 4, 2021, 8:25am UTC](https://discuss.elastic.co/t/eck-operator-stuck-in-applyingchanges-state-after-upgrade-to-1-4-0/265726/5 "2021-03-04T08:25:52Z")

</div>

> [@barracuda-charles](#):
>
> Also, I have a yaml file that I apply to edit the cluster -- should I keep the annotation in that yaml file?

That should not be necessary. As long as you use `kubectl apply` the content of that file will be merged with the state on the API server and the annotation should be preserved.

I am honestly a bit baffled by this error, given that you say there is nothing of interest in the operator logs. Can you check if you have the following log statement in the logs:

`Resource was previously reconciled by incompatible controller version and missing annotation, adding annotation`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 1, 2021, 8:26am UTC](https://discuss.elastic.co/t/eck-operator-stuck-in-applyingchanges-state-after-upgrade-to-1-4-0/265726/6 "2021-04-01T08:26:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.