Kibana can't sign in

Elastic Stack Elasticsearch
1

Kibana is not accepting the default elasticsearch-master-credentials

The following ELK stack configuration is all version 8.5.1 running in a single node cluster:

pod/elasticsearch-master-0       1/1     Running   0          28m
pod/logstash-0                   1/1     Running   0          28m
pod/elk-filebeat-j78fx           1/1     Running   0          28m
pod/elk-kibana-8596dcf6c-rdpzt   1/1     Running   0          25m

NAME                                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
service/logstash-headless               ClusterIP   None           <none>        9600/TCP            28m
service/elasticsearch-master-headless   ClusterIP   None           <none>        9200/TCP,9300/TCP   28m
service/logstash                        ClusterIP   10.43.12.129   <none>        5044/TCP,8080/TCP   28m
service/elasticsearch-master            ClusterIP   10.43.206.4    <none>        9200/TCP,9300/TCP   28m
service/elk-kibana                      ClusterIP   10.43.185.0    <none>        5601/TCP            25m

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/elk-filebeat   1         1         1       1            1           <none>          28m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/elk-kibana   1/1     1            1           25m

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/elk-kibana-8596dcf6c   1         1         1       25m

NAME                                    READY   AGE
statefulset.apps/elasticsearch-master   1/1     28m
statefulset.apps/logstash               1/1     28m

I port forward the kibana service and receive the Welcome to Elastic log on screen when connecting to http://www.localhost:5602/login

To log in I utilize the default elasticsearch-master-credentials username and password via

echo $(kubectl get secrets -n analytics elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d)
elastic
echo $(kubectl get secrets -n analytics elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d)
XSVoEFBANicJNZ3q

When attempting to log in, I receive the Username or password is incorrect. Please try again. error message and see the below in the elasticsearch POD log.

{"@timestamp":"2023-07-24T19:15:20.922Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master-0][system_critical_read][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","trace.id":"87d4f012851ecf678aa090d8cd662881","elasticsearch.cluster.uuid":"qti4HuKWS7ewy2HWIlu9KQ","elasticsearch.node.id":"16fM8XeXQ0KiY0uCtnetKg","elasticsearch.node.name":"elasticsearch-master-0","elasticsearch.cluster.name":"elasticsearch"}

Kibana:

[2023-07-24T19:15:20.844+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic)
2

Additional tests.

Check the decoded data of the elasticsearch-master-certs which contained:

ca.crt
tls.crt
tls.key

Matched the credentials at /usr/share/elasticsearch/config/certs within the elasticsearch container located. Both credential sets match.

Setup a port-forward for to access port 9200 of the elasticsearch container locally:

kubectl port-forward service/elasticsearch-master 9200:9200 -n analytics
Forwarding from 127.0.0.1:9200 -> 9200
Forwarding from [::1]:9200 -> 9200
Handling connection for 9200

Yet curl fails to get the cluster health so my issue appears to be with Elasticsearch? I can paste the full Elasticsearch logs here if they are of any benefit.

$curl -X GET "https://localhost:9200/_cluster/health" --key esearch_tls.crt -k -u elastic:XSVoEFBANicJNZ3q --verbose
vagrant@dd-ubuntu2204-sscld120npwd-a4f08c:~$ curl -X GET "https://localhost:9200/_cluster/health" --key esearch_tls.crt -k -u elastic:XSVoEFBANicJNZ3q --verbose
Note: Unnecessary use of -X or --request, GET is already inferred.
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1'
*   Trying 127.0.0.1:9200...
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=elasticsearch-master
*  start date: Jul 24 19:09:51 2023 GMT
*  expire date: Jul 23 19:09:51 2024 GMT
*  issuer: CN=elasticsearch-ca
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'elastic'
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /_cluster/health HTTP/1.1
> Host: localhost:9200
> Authorization: Basic ZWxhc3RpYzpYU1ZvRUZCQU5pY0pOWjNx
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
< content-type: application/json;charset=utf-8
< content-length: 495
< 
* Connection #0 to host localhost left intact
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_cluster/health]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_cluster/health]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
3

Turns out the issue was on my end with the secret generation being overwritten. All works now.