After Update from Kibana 7.x to Kibana 8.9 =>security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&f

Elastic Stack Kibana
1

Hello,
we updated yesterday our elasticsearch-stack to 8.9.0.
Now, kibana don't start and i found this error in /var/log/messages

FATAL  ResponseError: security_exception
Aug 17 10:42:58 elasticserver kibana[342912]: FATAL  ResponseError: security_exception
Aug 17 10:42:58 elasticserver kibana[342912]: #011Root causes:
Aug 17 10:42:58 elasticserver kibana[342912]: #011Root causes:
Aug 17 10:42:58 elasticserver kibana[342912]: #011#011security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&flat_settings=true]
Aug 17 10:42:58 elasticserver systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 10:42:58 elasticserver systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 17 10:42:58 elasticserver kibana[342912]: #011#011security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&flat_settings=true]
Aug 17 10:42:58 elasticserver systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 10:42:58 elasticserver systemd[1]: kibana.service: Failed with result 'exit-code'.

Okay.. i try to search something more about this error:
a) I've the following settings in kibana.yml:

elasticsearch.username: "kibana_system"
elasticsearch.password: "secure"

So i tried to reset the password for kibana_system => without success
I read in an other topic, that kibana_system is the wrong user, so i tried the user 'kibana' with new password => without success
b) In the instruction (Configure security in Kibana | Kibana Guide [8.9] | Elastic)
it seems that i don't need to set elasticsearch.username?? But without kibana starts up, but don't reach the elasticsarch-cluster.

I've set:

server.publicBaseUrl
elasticsearch.hosts (to https://xyz:9200)
server.name
elasticsearch.ssl.certificateAuthorities
xpack.security.encryptionKey: 'randomsecurekey'
xpack.encryptedSavedObjects.encryptionKey: 'securekey'
xpack.fleet.enabled: false

So i'm a little bit confused, what to do.. the 8.9 instruction say nothing about the kibana user setting and don't know, what more i could try.

Thank you for reading :wink:
Wolfgang

2

Nope, kibana_system is the correct user and you still need it, but you need to double check that you use correct password in elasticsearch.password.

Alternatively, you can create a service account token for kibana/system service account and configure Kibana to use it (elasticsearch.serviceAccountToken) instead of elasticsearch.username and elasticsearch.password pair.

3

Hi

We just did the upgrade from 7.16 to 8.9 too last week.

You need to check the upgrade assistant before upgrading:

Screenshot from our own upgrade guide:
image

The message was taken from the upgrade assistant, where the "Learn more" link points to:

Elasticsearch logs should indicate something that elasticsearch.username is deprecated for 8.9.

Good luck!
Christof

4

Thank you for the answers. I tried another one to use 'kibana_system' for kibana:
Step1 - Reset the kibana_system password:

 ./elasticsearch-reset-password -u kibana_system
This tool will reset the password of the [kibana_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y


Password for the [kibana_system] user successfully reset.
New value: EbmPfDZJ7W-7K7xH=OAL

Step2 - Editing the kibana.yml

elasticsearch.username: "kibana_system"
elasticsearch.password: "EbmPfDZJ7W-7K7xH=OAL"

Step3 - Restart Kibana:
It fails with this lines in /var/log/messsages

Aug 17 10:42:58 kvmua384 kibana[342912]: FATAL  ResponseError: security_exception
Aug 17 10:42:58 kvmua384 kibana[342912]: FATAL  ResponseError: security_exception
Aug 17 10:42:58 kvmua384 kibana[342912]: #011Root causes:
Aug 17 10:42:58 kvmua384 kibana[342912]: #011Root causes:
Aug 17 10:42:58 kvmua384 kibana[342912]: #011#011security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&f

We have used the upgrade guide and so we used the kibana_system user ...

5

That's weird. Can you check that you don't have a conflicting kibana_system password in Kibana Keystore: Secure settings | Kibana Guide [8.11] | Elastic?

Also try to enable verbose logs in Kibana, maybe it'd reveal more information: logging.root.level: debug?

6

Thank you very much, i've solved the problem:
I took the kibana.yml.rpmnew and wrote my old settings in it, then i copied it to kibana.yml. Now all it's okay.
My failure to ignore the rpmnew file, i don't know what setting, but now it runs.

7

Awesome, glad you solved it!