After K8s cluster was turned off for night Kibna "Username or password is incorrect"

Elastic Stack Kibana
1

I have a problem with Kibana, when my Kubernetes cluster shuts down in the evening and turns back on in the morning, then I can't log in with the credentials that were valid yesterday. And when I delete the pod and it is rebuilt, the credentials are valid again, what's the problem?

Kibana is attached to Elasticsearch and to login to Kibana I use Elasticsearch login and password and they are present in the cluster even after shutting down and restarting the cluster. and at the same time, when logging in to Elasticsearch itself at localhost:9200, I get access correctly, with the same credentials that I tried before on Kiban.
my log in kibana when I try vloginitis:
// {"log":"[2023-06-28T18:18:44.023+00:00][INFO ][plugins.security.routes] Logging in with provider \"basic\" (basic)\n","stream":"stdout","time":"2023-06-28T18:18:44.024420008Z"}'

2

How are you creating your cluster on Kubernetes? Does all your Elasticsearch nodes have persistent volume?

Also, how are you recreating it?

3

Kibana hasn't persistent volume but Elastic has it. For recreate i just delete the pod and when it is recreated I can login to Kibana

4

I use official helm chart of ELK stack

5

Which version of the Stack are you using? Elastic does not public helm charts anymore.

Also, share the Elasticsearch logs when you start it and have this error.

6

artifacthub elasticsearch and kibana version 8.5.1

i have the same stack on another cluster but it is one without turning off at night and I don't have this problem.

I don't see any logs about this problem in the logs of Elasticsearch.

7

Without any evidence it is pretty hard to troubleshoot it.

Next time you have this issue please get the logs for both Kibana and your Elasticsearch nodes.

If you can't login with the same credentials something change.

How are you configuring the credentials for your Kibana instance? You said you do not have a persistent volume for Kibana, but how will the new pod use the same credentials? I do not use k8s, so not sure how you would do that.

I would recommend that you set a persistent volume for Kibana as well.

8

Into values.yml of kibana helm chart is setted

''elasticsearchHosts: "elasticsearch-master:9200"
elasticsearchCertificateSecret: elasticsearch-master-certs
elasticsearchCertificateAuthoritiesFile: ca.crt
elasticsearchCredentialSecret: elasticsearch-master-credentials''

and creds saved into secrets of k8s

9

And I don't have this issue today, but I had this issue for two latest months. I tried to fix it yesterday with chatGPT. Maybe something help to fix it. I will try tomorrow again. I let you know about it tomorrow.

10

I cache this error again

image
image686×605 20.5 KB

I got just the next log Into logs Kibana when try to login:

[2023-07-17T11:25:55.363+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)

Into Elasticsearch logs, I don't see any logs which show trying to login

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.