"Resource was created with older version of operator, will not take action" cause ES cluster pods to stuck

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I am facing an issue with the Elasticsearch (w/o kibana or apm) cluster deployment with the custom operator image compiled and built locally (using latest master without any changes) and tried to deploy a simple Elasticsearch yaml, the statefulset pod is stuck in the Init:1/2 state.

I am seeing the following in the operator log which might be the issue here:

{"level":"info","ts":1567096658.9352784,"logger":"annotation","msg":"Resource was created with older version of operator, will not take action","controller_version":"0.0.0","resource_controller_version":"0.0.0","namespace":"default","name":"quickstart"}

Here is the kubectl output for the pod:
default quickstart-es-test-0 0/1 Init:1/2 0 4m51s 172.17.0.6 minikube <none> <none>

The version: 7.3.0 is specified in the elasticsearch yaml and using following operaor image: sarjesingh/eck-operator:0.10.0-SNAPSHOT-d07c6f08

Please let me know if you'd need me to debug into something to see the issue.

Note: I am able to deploy elasticsearch cluster fine from the quickstart guide.

2

The weird thing is that when reverted operator image back to docker.elastic.co/eck/eck-operator:0.9.0 and re-created elastic cluster, it works fine.

Not sure how custom image could create the container stuck with the controller/resource version mismatch unless I am missing some information from elasticsearch yaml itself.

3

Filed issue https://github.com/elastic/cloud-on-k8s/issues/1666