I recently upgraded to 1.4.0 and the operator is stuck in the ApplyingChanges state. I expanded a few volumes on data nodes which caused the cluster to enter a yellow state temporarily however the operator stayed green the whole time. It has been over 12 hours since the upgrade to 1.4.0 (and the cluster has been in a green state since then:
$ k get es
NAME HEALTH NODES VERSION PHASE AGE
search green 63 6.8.6 ApplyingChanges 289d
I don't see anything relevant in the logs for the operator:
{"log.level":"info","@timestamp":"2021-02-28T14:25:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:27:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:29:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:31:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:33:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:35:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-28T14:37:52.632Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
It looks like the es cluster has an invalid annotation:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
annotations:
common.k8s.elastic.co/controller-version: 0.0.0-UNKNOWN
This was an upgrade from 1.2.x to 1.4.0 -- should I try downgrading to 1.3.x and then back to 1.4.0 to resolve the inconsistency, or perhaps edit the annotation to something suitable?
When I change the manifest I see the following in the log:
{"log.level":"info","@timestamp":"2021-02-28T20:00:51.333Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","iteration":3,"namespace":"default","es_name":"search"}
{"log.level":"info","@timestamp":"2021-02-28T20:00:51.333Z","log.logger":"annotation","message":"Resource was created with older version of operator, will not take action","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","controller_version":"1.4.0","resource_controller_version":"0.0.0-UNKNOWN","namespace":"default","name":"search"}
{"log.level":"info","@timestamp":"2021-02-28T20:00:51.333Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","iteration":3,"namespace":"default","es_name":"search","took":0.000139559}