# After Update from Kibana 7.x to Kibana 8.9 =\>security\_exception: unable to authenticate user \[kibana\_system\] for REST request \[/\_cluster/settings?include\_defaults=true&f

**URL:** <https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984>\
**Category:** Kibana\
**Tags:** elastic-stack-security\
**Created:** [August 17, 2023, 9:10am UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984 "2023-08-17T09:10:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wolfgang\_Winter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfgang_winter/32/124689_2.png) [@Wolfgang\_Winter](https://discuss.elastic.co/u/Wolfgang_Winter)\
**Post date:** [August 17, 2023, 9:10am UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984/1 "2023-08-17T09:10:48Z")

</div>

Hello,  
we updated yesterday our elasticsearch-stack to 8.9.0.  
Now, kibana don't start and i found this error in /var/log/messages

```auto
FATAL ResponseError: security_exception
Aug 17 10:42:58 elasticserver kibana[342912]: FATAL ResponseError: security_exception
Aug 17 10:42:58 elasticserver kibana[342912]: #011Root causes:
Aug 17 10:42:58 elasticserver kibana[342912]: #011Root causes:
Aug 17 10:42:58 elasticserver kibana[342912]: #011#011security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&flat_settings=true]
Aug 17 10:42:58 elasticserver systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 10:42:58 elasticserver systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 17 10:42:58 elasticserver kibana[342912]: #011#011security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&flat_settings=true]
Aug 17 10:42:58 elasticserver systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 10:42:58 elasticserver systemd[1]: kibana.service: Failed with result 'exit-code'.

```

Okay.. i try to search something more about this error:  
a) I've the following settings in kibana.yml:

```auto
elasticsearch.username: "kibana_system"
elasticsearch.password: "secure"

```

So i tried to reset the password for kibana\_system =\> without success  
I read in an other topic, that kibana\_system is the wrong user, so i tried the user 'kibana' with new password =\> without success  
b) In the instruction ([Configure security in Kibana | Kibana Guide [8.9] | Elastic](https://www.elastic.co/guide/en/kibana/8.9/using-kibana-with-security.html))  
it seems that i don't need to set elasticsearch.username?? But without kibana starts up, but don't reach the elasticsarch-cluster.

I've set:

```auto
server.publicBaseUrl
elasticsearch.hosts (to https://xyz:9200)
server.name
elasticsearch.ssl.certificateAuthorities
xpack.security.encryptionKey: 'randomsecurekey'
xpack.encryptedSavedObjects.encryptionKey: 'securekey'
xpack.fleet.enabled: false

```

So i'm a little bit confused, what to do.. the 8.9 instruction say nothing about the kibana user setting and don't know, what more i could try.

Thank you for reading 😉  
Wolfgang

---

<div class="post-metadata">

**Author:** ![azasypkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azasypkin/32/42216_2.png) [@azasypkin](https://discuss.elastic.co/u/azasypkin)\
**Post date:** [August 17, 2023, 9:32am UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984/2 "2023-08-17T09:32:40Z")

</div>

> [@Wolfgang\_Winter](#):
>
> read in an other topic, that kibana\_system is the wrong user

Nope, `kibana_system` **is** the correct user and you still need it, but you need to [double check](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html#set-built-in-user-passwords) that you use correct password in `elasticsearch.password`.

Alternatively, you can create a [service account token](https://www.elastic.co/guide/en/elasticsearch/reference/current/service-tokens-command.html) for `kibana/system` service account and configure Kibana to use it (`elasticsearch.serviceAccountToken`) instead of `elasticsearch.username` and `elasticsearch.password` pair.

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [August 17, 2023, 12:01pm UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984/3 "2023-08-17T12:01:13Z")

</div>

Hi

We just did the upgrade from 7.16 to 8.9 too last week.

You need to check the upgrade assistant before upgrading:

> **[Upgrade Assistant | Kibana Guide \[7.17\] | Elastic](https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html)**
>
> Kibana provides you with several options to share \*Discover\* saved searches, dashboards, \*Visualize Library\* visualizations, and \*Canvas\* workpads with others, or on a website.

Screenshot from our own upgrade guide:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/8/b86b0b8827e1a6353f340a2e21dc681613750855.png)

The message was taken from the upgrade assistant, where the "Learn more" link points to:

> **[Service accounts | Elasticsearch Guide \[7.17\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/service-accounts.html)**

Elasticsearch logs should indicate something that elasticsearch.username is deprecated for 8.9.

Good luck!  
Christof

---

<div class="post-metadata">

**Author:** ![Wolfgang\_Winter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfgang_winter/32/124689_2.png) [@Wolfgang\_Winter](https://discuss.elastic.co/u/Wolfgang_Winter)\
**Post date:** [August 17, 2023, 2:43pm UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984/4 "2023-08-17T14:43:53Z")

</div>

Thank you for the answers. I tried another one to use 'kibana\_system' for kibana:  
Step1 - Reset the kibana\_system password:

```auto
 ./elasticsearch-reset-password -u kibana_system
This tool will reset the password of the [kibana_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y

Password for the [kibana_system] user successfully reset.
New value: EbmPfDZJ7W-7K7xH=OAL

```

Step2 - Editing the kibana.yml

```auto
elasticsearch.username: "kibana_system"
elasticsearch.password: "EbmPfDZJ7W-7K7xH=OAL"

```

Step3 - Restart Kibana:  
It fails with this lines in /var/log/messsages

```auto
Aug 17 10:42:58 kvmua384 kibana[342912]: FATAL ResponseError: security_exception
Aug 17 10:42:58 kvmua384 kibana[342912]: FATAL ResponseError: security_exception
Aug 17 10:42:58 kvmua384 kibana[342912]: #011Root causes:
Aug 17 10:42:58 kvmua384 kibana[342912]: #011Root causes:
Aug 17 10:42:58 kvmua384 kibana[342912]: #011#011security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&f

```

We have used the upgrade guide and so we used the kibana\_system user ...

---

<div class="post-metadata">

**Author:** ![azasypkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azasypkin/32/42216_2.png) [@azasypkin](https://discuss.elastic.co/u/azasypkin)\
**Post date:** [August 17, 2023, 3:11pm UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984/5 "2023-08-17T15:11:51Z")

</div>

> [@Wolfgang\_Winter](#):
>
> `Aug 17 10:42:58 kvmua384 kibana[342912]: #011#011security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&f`

That's weird. Can you check that you don't have a conflicting `kibana_system` password in Kibana Keystore: [Secure settings | Kibana Guide [8.11] | Elastic](https://www.elastic.co/guide/en/kibana/current/secure-settings.html)?

Also try to enable verbose logs in Kibana, maybe it'd reveal more information: `logging.root.level: debug`?

---

<div class="post-metadata">

**Author:** ![Wolfgang\_Winter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfgang_winter/32/124689_2.png) [@Wolfgang\_Winter](https://discuss.elastic.co/u/Wolfgang_Winter)\
**Post date:** [August 21, 2023, 10:13am UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984/6 "2023-08-21T10:13:23Z")

</div>

> [@azasypkin](#):
>
> logging.root.level: debug

Thank you very much, i've solved the problem:  
I took the kibana.yml.rpmnew and wrote my old settings in it, then i copied it to kibana.yml. Now all it's okay.  
My failure to ignore the rpmnew file, i don't know what setting, but now it runs.

---

<div class="post-metadata">

**Author:** ![azasypkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azasypkin/32/42216_2.png) [@azasypkin](https://discuss.elastic.co/u/azasypkin)\
**Post date:** [August 21, 2023, 10:24am UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984/7 "2023-08-21T10:24:36Z")

</div>

Awesome, glad you solved it!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2023, 10:25am UTC](https://discuss.elastic.co/t/after-update-from-kibana-7-x-to-kibana-8-9-security-exception-unable-to-authenticate-user-kibana-system-for-rest-request-cluster-settings-include-defaults-true-f/340984/8 "2023-09-18T10:25:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.