ECK Fail to update status field of elasticsearch cluster on EKS 1.19

Hi, i am trying to get a cluster running on EKS 1.19 to sucessfully create an elasticsearchcluster using the ECK operator 1.6

the cluster gets created just fine when i create the elasticsearch resource and i can see the status is green when i just curl the service port 9200

The operator fails to update the status of the cluster though so it just looks like

jaeger                                      11m

Logs from the operator show a not found error

elastic-operator-0 manager {"log.level":"error","@timestamp":"2021-07-21T09:00:05.214Z","log.logger":"manager.eck-operator.controller.elasticsearch-controller","message":"Reconciler error","service.version":"1.6.0+8326ca8a","service.type":"eck","ecs.version":"1.4.0","name":"jaeger","namespace":"jaeger-server","error":" \"jaeger\" not found","errorCauses":[{"error":" \"jaeger\" not found"}],"error.stack_trace":"*Controller).processNextWorkItem\n\t/go/pkg/mod/\*Controller).Start.func1.2\n\t/go/pkg/mod/\\n\t/go/pkg/mod/\\n\t/go/pkg/mod/\\n\t/go/pkg/mod/\\n\t/go/pkg/mod/\\n\t/go/pkg/mod/\\n\t/go/pkg/mod/"}

and digging into the CloudWatch apiserver logs i can find indeed the request being authorized but returning a 404 not found

@ingestionTime | 1626858005606
-- | --
@log | 542491017901:/aws/eks/stag101-us-east-1/cluster
@logStream | kube-apiserver-audit-2b59cc97274ecf922e440f703cd8faaa
@message | {"kind":"Event","apiVersion":"","level":"Metadata","auditID":"d1e5d737-faa6-479c-88e2-0aa51b2a3afb","stage":"ResponseComplete","requestURI":"/apis/","verb":"update","user":{"username":"system:serviceaccount:elastic-system:elastic-operator","uid":"c00d05ae-68eb-4194-af50-d9445315b45f","groups":["system:serviceaccounts","system:serviceaccounts:elastic-system","system:authenticated"]},"sourceIPs":[""],"userAgent":"elastic-operator/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"elasticsearches","namespace":"jaeger-server","name":"jaeger","apiGroup":"","apiVersion":"v1","subresource":"status"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2021-07-21T09:00:05.213978Z","stageTimestamp":"2021-07-21T09:00:05.214150Z","annotations":{"":"system:serviceaccount:elastic-system:elastic-operator","":"allow","":"RBAC: allowed by ClusterRoleBinding \"elastic-operator\" of ClusterRole \"elastic-operator\" to ServiceAccount \"elastic-operator/elastic-system\""}}
@timestamp | 1626858005339 | system:serviceaccount:elastic-system:elastic-operator | allow | RBAC: allowed by ClusterRoleBinding "elastic-operator" of ClusterRole "elastic-operator" to ServiceAccount "elastic-operator/elastic-system"
apiVersion |
auditID | d1e5d737-faa6-479c-88e2-0aa51b2a3afb
kind | Event
level | Metadata
objectRef.apiGroup |
objectRef.apiVersion | v1 | jaeger
objectRef.namespace | jaeger-server
objectRef.resource | elasticsearches
objectRef.subresource | status
requestReceivedTimestamp | 2021-07-21T09:00:05.213978Z
requestURI | /apis/
responseStatus.code | 404
responseStatus.reason | NotFound
responseStatus.status | Failure
sourceIPs.0 |
stage | ResponseComplete
stageTimestamp | 2021-07-21T09:00:05.214150Z
user.groups.0 | system:serviceaccounts
user.groups.1 | system:serviceaccounts:elastic-system
user.groups.2 | system:authenticated
user.uid | c00d05ae-68eb-4194-af50-d9445315b45f
user.username | system:serviceaccount:elastic-system:elastic-operator
userAgent | elastic-operator/v0.0.0 (linux/amd64) kubernetes/$Format
verb | update

while i can see successful update requests to the non status URI /apis/

a bit more testing

kind 1.19

with the operator deployed and the quickstart ES it works just fine , using kubectl proxy i can get

  • curl -k -s -X GET -H "Accept: application/json, */*" -H "Content-Type: application/json" "" -> 200 Works

on EKS 1.19

  • curl -k -s -X GET -H "Accept: application/json, */*" -H "Content-Type: application/json" "" -> 200 Worls
  • curl -k -s -X GET -H "Accept: application/json, */*" -H "Content-Type: application/json" "" -> 404 not found
  • curl -k -s -X GET -H "Accept: application/json, */*" -H "Content-Type: application/json" "" -> 200 Works


How the operator has been installed ? Is it an upgrade from a previous release ?

Could you provide the result of the following command: kubectl get crds -o yaml | grep -n -A 2 -B 2 'subresources:'


Hi, the operator was installed by rendering the helm chart to manifests and than applied via ArgoCD. It was a first install but i plan to upgrade it by re-rendering the same way and letting Argo apply it ( is that ok ? )

    helm template elastic elastic/eck-operator \
        --version 1.6.0 --kube-version v1.19.0 --dry-run --include-crds \
        --namespace elastic-system \
        --set=installCRDs=true \
        --set=webhook.enabled=true \
        --set=config.logVerbosity="0" \
        --set=config.metricsPort="0" \
        --set=config.caValidity="87600h" \
        --set=config.caRotateBefore="240h" \
        --set=config.certificatesRotateBefore="240h" \
        --set=config.kubeClientTimeout="60s" \
        --set=config.elasticsearchClientTimeout="180s" \
        --set=podMonitor.enabled=false \
        --set=global.createOperatorNamespace=false \
        --set=global.kubeVersion=1.19.0 \

kubectl get crds -o yaml | grep -n -A 2 -B 2 'subresources:' gives me no output

checking the upstream all-in-one i can see i should be getting

$ cat | grep -n -A 2 -B 2 'subresources:'
42-    singular: elasticsearch
43-  scope: Namespaced
44:  subresources:
45-    status: {}
46-  validation:

but when i check the manifests i rendered from helm i can see is not there ( so not jus tlive but also in my checked in manifests)

I am gonna look into this to figure out what happened.


FYI this was my fault, i am back from holiday and realized my jsonnet was removing those fields by mistake.

thanks @michael.morello for pointing me in the right direction

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.