Kibana Authentication attempt failed: UNEXPECTED_SESSION_ERROR

Elastic Stack Kibana
1

Hello!

First time posting and i have been using the ELK stack for quite a while now.
I've encountered an issue that i haven't found much about in the forums or elsewhere and it's related to the security when it comes to the authentication of users.
I have a self-managed ELK cluster and sitting on top of the kibana i have an apache server with a reverse proxy for identification on my organization. I have a basic license on kibana but to my understanding it should be working (and it was in the past).
I'm only using the kibana authentication with basic and anonymous users and it's only happening with the guest user (anonymous). There's a session timeout set up but it's only after 8 hours and it usually kicks you out of Kibana after around 3 hours.
Apache is not outputting an error and the log from the kibana.log is as follows:

[2024-11-01T12:12:01.758-03:00][INFO ][plugins.security.authentication] Performing login attempt with "anonymous1" provider.
[2024-11-01T12:12:01.793-03:00][INFO ][plugins.security.authentication] Login attempt with "anonymous1" provider succeeded (requires redirect: true).
[2024-11-01T12:16:51.081-03:00][INFO ][plugins.security.authentication] Performing login attempt with "anonymous1" provider.
[2024-11-01T12:16:51.112-03:00][INFO ][plugins.security.authentication] Login attempt with "anonymous1" provider succeeded (requires redirect: true).
[2024-11-01T12:31:51.836-03:00][INFO ][plugins.fleet] Running Fleet Usage telemetry send task
[2024-11-01T12:39:50.234-03:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"num_host_urls":0}}
[2024-11-01T12:52:32.688-03:00][INFO ][plugins.security.authentication] Performing login attempt with "anonymous1" provider.
[2024-11-01T12:52:32.736-03:00][INFO ][plugins.security.authentication] Login attempt with "anonymous1" provider succeeded (requires redirect: true).
[2024-11-01T13:09:52.399-03:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"num_host_urls":0},"license_issued_to:}
[2024-11-01T13:31:52.974-03:00][INFO ][plugins.fleet] Running Fleet Usage telemetry send task
[2024-11-01T13:39:54.461-03:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"num_host_urls":0},"license_issued_to"}
[2024-11-01T14:32:11.948-03:00][ERROR][plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR
[2024-11-01T14:32:53.413-03:00][ERROR][plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR
[2024-11-01T14:36:37.082-03:00][INFO ][plugins.security.authentication] Performing login attempt with "anonymous1" provider.
[2024-11-01T14:36:37.123-03:00][INFO ][plugins.security.authentication] Login attempt with "anonymous1" provider succeeded (requires redirect: true).
[2024-11-01T14:40:01.864-03:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"num_host_urls":0},"license_issued_to"}
[2024-11-01T14:40:19.212-03:00][ERROR][plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR
[2024-11-01T14:47:05.987-03:00][ERROR][plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR
[2024-11-01T14:47:06.119-03:00][ERROR][plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR

I don't really have any more details on the error since this as much as i can get to this, everything else in the cluster seems to be working fine.

My kibana.yml configuration related to the kibana.log:


server.port: 5601

server.host: "localhost"

server.name: "Prod"

elasticsearch.hosts:
  - https://(elasticnode1):9200
  - https://(elasticnode2):9200
  - https://(elasticnode3)9200

pid.file: /route/to/pid/kibana.pid


logging:
  appenders:
    file:
      type: file
      fileName: /ELK/kibana-8.15.0/logs/kibana.log
      layout:
        type: pattern
  root:
    appenders: [default, file]

xpack.security.session.idleTimeout: "8h"
xpack.security.session.lifespan: "30d"

elasticsearch.ssl.certificateAuthorities: "/route/to/cert/elasticsearch-ca.pem"

xpack.security.authc.providers:
  basic.basic1: 
    order: 1
  anonymous.anonymous1:
    order: 0

kibana.autocompleteTimeout: 1000
kibana.autocompleteTerminateAfter: 100000

monitoring.kibana.collection.enabled: false

The anonymous user is stored in the kibana keystore and i have tried changing the password to no effect.
Thanks in advance!

2

Hi @seb_l ,

At first look your kibana.yml seems to be correct. Could you confirm that kick out after 3h is only for anonymous user and it last expected 8h when you authenticate explicitly with basic authentication?

Did you try to explicitly set

    credentials:
      username: "anonymous_service_account"
      password: "anonymous_service_account_password"

in kibana.yml and see if the issue persist?

Regards, Dima

3

Could you confirm that kick out after 3h is only for anonymous user and it last expected 8h when you authenticate explicitly with basic authentication?

Correct!
I tried setting up the credentials in the kibana.yml settings and i'm still getting the same issue. I even went ahead and created a new user with a slightly changed name, different password, same role and the issue still persists if you opt in for the option "Continue as a guest"

4

Completely forgot to reply but i found the solution to it.
I had another instance of Kibana running for more availability and turns out that the keystores were not synced and was missing the username and password for the anonymous user.
Adding the same ones as the other keystore fixed the problem.

Although wished the error was a bit more detailed explaining why this was happening in the first place.