# Reconciler error: auth secret key default-quickstart-kibana-user doesn't exist

**URL:** <https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [June 14, 2022, 10:40am UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136 "2022-06-14T10:40:39Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![oliverrahner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oliverrahner/32/107003_2.png) [@oliverrahner](https://discuss.elastic.co/u/oliverrahner)\
**Post date:** [June 14, 2022, 10:40am UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/1 "2022-06-14T10:40:39Z")

</div>

I am currently upgrading my sandbox ECK installation from 7.14.0 to 8.2.2, via 7.17, as required.

I managed to finish this for Elasticsearch, but during the 7.14 -\> 7.17 step for Kibana, I stumbled across an issue.

The operator seems to try to do the update, but bails out with the following message:

```auto
{
   "log.level":"error",
   "@timestamp":"2022-06-14T10:28:35.377Z",
   "log.logger":"manager.eck-operator.controller.kibana-controller",
   "message":"Reconciler error",
   "service.version":"2.2.0+02f250eb",
   "service.type":"eck",
   "ecs.version":"1.4.0",
   "name":"quickstart",
   "namespace":"default",
   "error":"auth secret key default-quickstart-kibana-user doesn't exist",
   "errorCauses":[
      {
         "error":"auth secret key default-quickstart-kibana-user doesn't exist",
         "errorVerbose":"auth secret key default-quickstart-kibana-user doesn't exist\ngithub.com/elastic/cloud-on-k8s/pkg/controller/association.ElasticsearchAuthSettings\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/association/conf.go:96\ngithub.com/elastic/cloud-on-k8s/pkg/controller/kibana.NewConfigSettings\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/kibana/config_settings.go:139\ngithub.com/elastic/cloud-on-k8s/pkg/controller/kibana.(*driver).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/kibana/driver.go:149\ngithub.com/elastic/cloud-on-k8s/pkg/controller/kibana.(*ReconcileKibana).doReconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/kibana/controller.go:196\ngithub.com/elastic/cloud-on-k8s/pkg/controller/kibana.(*ReconcileKibana).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/kibana/controller.go:161\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"
      }
   ],
   "error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227"
}

```

I verified that the secret does exist in the cluster:

```auto
# kubectl get secret default-quickstart-kibana-user
NAME TYPE DATA AGE
default-quickstart-kibana-user Opaque 2 257d

```

---

<div class="post-metadata">

**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\
**Post date:** [June 14, 2022, 1:39pm UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/2 "2022-06-14T13:39:17Z")

</div>

This error message leaves room for improvement. Sorry about that. What is actually meant is that a key inside the secret `quickstart-kibana-user` is missing.

Maybe you can check what key the controller is looking for by running this:

```nohighlight
kubectl get kb quickstart -o go-template='{{index .metadata.annotations "association.k8s.elastic.co/es-conf"}}'

```

and comparing it with the contents of `default-quickstart-kibana-user` will maybe help us shed a bit of light on what is exactly missing.

---

<div class="post-metadata">

**Author:** ![oliverrahner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oliverrahner/32/107003_2.png) [@oliverrahner](https://discuss.elastic.co/u/oliverrahner)\
**Post date:** [June 14, 2022, 3:43pm UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/3 "2022-06-14T15:43:55Z")

</div>

I'm not completely sure what to look for, but this is what I found:

```auto
# kubectl get kb quickstart -o go-template='{{index .metadata.annotations "association.k8s.elastic.co/es-conf"}}' | jq
{
  "authSecretName": "quickstart-kibana-user",
  "authSecretKey": "default-quickstart-kibana-user",
  "caCertProvided": true,
  "caSecretName": "quickstart-kb-es-ca",
  "url": "https://quickstart-es-http.default.svc:9200",
  "version": "8.2.2"
}

```

```auto
# kubectl get secret default-quickstart-kibana-user -o jsonpath={.data.hash} | base64 -d
{PBKDF2_STRETCH}10000$Fq6....

```

---

<div class="post-metadata">

**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\
**Post date:** [June 14, 2022, 4:31pm UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/4 "2022-06-14T16:31:10Z")

</div>

Based on the output from the annotation there should be a secret called `quickstart-kibana-user` that should contain a key called `default-quick start-kibana-user` if that is not there then the association between Elasticsearch and Kibana is not established correctly. What does the status sub resource on the Kibana say? Are there any Kubernetes events of interest maybe? Anything else in the logs?

---

<div class="post-metadata">

**Author:** ![oliverrahner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oliverrahner/32/107003_2.png) [@oliverrahner](https://discuss.elastic.co/u/oliverrahner)\
**Post date:** [June 14, 2022, 4:45pm UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/5 "2022-06-14T16:45:17Z")

</div>

Phew, ECK is my Kubernetes learning project 😉 I'll try to unroll your questions:  
There _is_ a secret called `quickstart-kibana-user`, but it does **not** seem to contain the requested key:

```auto
{
  "hash": "e1BCS0RGMl...",
  "name": "ZGVmYX...",
  "serviceAccount": "ZWxhc3RpYy9raWJhbmE=",
  "token": "QUFFQ..."
}

```

Status sub resource? Do you mean this?

```auto
# kubectl get kibanas.kibana.k8s.elastic.co
NAME HEALTH NODES VERSION AGE
quickstart red 7.15.0 257d

```

Status is red because of the incompatibility between Kibana 7.15.0 and Elasticsearch 8.2.2:

```auto
# kubectl logs --tail=2 quickstart-kb-56ffc5f89f-vvxb8
{"type":"log","@timestamp":"2022-06-14T08:05:58+00:00","tags":["error","savedobjects-service"],"pid":1217,"message":"This version of Kibana (v7.15.0) is incompatible with the following Elasticsearch nodes in your cluster: v8.2.2 @ quickstart-es-default-2.quickstart-es-default.default.svc/10.244.4.8:9200 (10.244.4.8), v8.2.2 @ quickstart-es-default-1.quickstart-es-default.default.svc/10.244.8.3:9200 (10.244.8.3), v8.2.2 @ quickstart-es-default-0.quickstart-es-default.default.svc/10.244.7.5:9200 (10.244.7.5)"}
{"type":"log","@timestamp":"2022-06-14T08:10:44+00:00","tags":["error","savedobjects-service"],"pid":1217,"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [default-quickstart-kibana-user] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}

```

Looking at this, I'm wondering why it first tells me that the versions aren't compatible, and then tells me it can't retrieve the versions?

K8s events:

```auto
# kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
110s Warning Unhealthy pod/quickstart-kb-56ffc5f89f-vvxb8 Readiness probe failed: HTTP probe failed with statuscode: 503
60m Warning ReconciliationError kibana/quickstart Reconciliation error: auth secret key default-quickstart-kibana-user doesn't exist
7m36s Warning ReconciliationError kibana/quickstart Reconciliation error: auth secret key default-quickstart-kibana-user doesn't exist

```

Nothing new I presume...  
Other than that, I am a bit out of ideas of where to look

Before the start of the upgrade, everything ran smoothly. But I did have some issues at the beginning of the process, regarding volume attachments that failed with a "Multi Attach Error". From what I learned of K8s to date, I don't think that could have anything to do with information from a secret getting (partially!) lost.

EDIT:  
I tried to put the value of secret `default-quickstart-kibana-user` as the value of that key under `quickstart-kibana-user`, but changes to the secret don't seem to be accepted, even though kubectl tells me it edited the secret:

```auto
# echo e1BCS... | read output;kubectl patch secret quickstart-kibana-user -p="{\"data\":{\"default-quickstart-kibana-user\": \"$output\"}}" -v=1
secret/quickstart-kibana-user patched

```

---

<div class="post-metadata">

**Author:** ![oliverrahner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oliverrahner/32/107003_2.png) [@oliverrahner](https://discuss.elastic.co/u/oliverrahner)\
**Post date:** [June 15, 2022, 7:22am UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/6 "2022-06-15T07:22:10Z")

</div>

I decided to remove the kibana instance and set it up again... thanks for your help.

---

<div class="post-metadata">

**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\
**Post date:** [June 15, 2022, 11:11am UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/7 "2022-06-15T11:11:45Z")

</div>

Yes removing and re-creating the Kibana instance should work in any case. I was still interested in the error you saw in case it points to a bug in the operator. But I was not able to reproduce your issue when upgrading a Elasticsearch/Kibana in the same way you did.

Just to give some additional background. We transitioned to using service account tokens instead of username/password for the internal communication between Kibana and Elasticsearch in 7.17.0 and you got somehow caught in the middle during your upgrade attempt. The error you saw as I understand it was a Kibana still expecting a username password combo in the secret and the operator already having replaced it with a service account token.

---

<div class="post-metadata">

**Author:** ![oliverrahner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oliverrahner/32/107003_2.png) [@oliverrahner](https://discuss.elastic.co/u/oliverrahner)\
**Post date:** [June 15, 2022, 2:07pm UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/8 "2022-06-15T14:07:40Z")

</div>

Thanks for elaborating.  
IIRC, I updated the operator (don't remember from what version, but definitely \<2) _after_ updating Elasticsearch, but _before_ updating Kibana, just because I forgot to update Kibana.  
Maybe that had something to do with it?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 13, 2022, 2:07pm UTC](https://discuss.elastic.co/t/reconciler-error-auth-secret-key-default-quickstart-kibana-user-doesnt-exist/307136/9 "2022-07-13T14:07:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.