# Failed to retrieve password hash for reserved user kibana\_system **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-security\ **Created:** [July 17, 2024, 10:45am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271 "2024-07-17T10:45:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Mireiabm](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@Mireiabm](https://discuss.elastic.co/u/Mireiabm)\ **Post date:** [July 17, 2024, 10:45am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/1 "2024-07-17T10:45:39Z")
hi, From one day to the next, my Elasticsearch stopped working without me doing anything, and this error appears:` [2024-07-11T09:22:37,457][ERROR][o.e.x.s.a.e.ReservedRealm] [prod-1] failed to retrieve password hash for reserved user [kibana_system]` What I did was change the password for kibana\_system, update it in the kibana.yml file, restart Elasticsearch, and it still doesn't work. The cluster status appears correct, and the password verification for kibana\_system also checks out fine. I don't know what else I can do. **They know what might be happening; everything seems fine.** ```auto Enter host password for user 'elastic': {"cluster_name":"prod","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":179,"active_shards":179,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}u index shard prirep state docs store dataset ip node .security-7 0 p STARTED 190 489.8kb 489.8kb Ip prod-1 ``` **I have changed and configured the new password for kibana\_system, the cluster is green, all shards are started. What else can I look at?** Elasticsearch Version 8.14.2 ubuntu 22.04 Thanks a lot! ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 18, 2024, 1:49am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/2 "2024-07-18T01:49:47Z")
> [@Mireiabm](#): > > [2024-07-11T09:22:37,457][ERROR][o.e.x.s.a.e.ReservedRealm] [prod-1] failed to retrieve password hash for reserved user [kibana\_system] Is there more to that error message? It is highly unlikely for the logs to print that message without additional details. ---
**Author:** ![Mireiabm](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@Mireiabm](https://discuss.elastic.co/u/Mireiabm)\ **Post date:** [July 18, 2024, 6:57am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/3 "2024-07-18T06:57:29Z")
```auto [2024-07-18T06:56:20,286][ERROR][o.e.x.s.a.e.ReservedRealm] [prod-1] failed to retrieve password hash for reserved user [kibana_system] org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:178) ~[?:?] at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:640) ~[?:?] at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?] at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?] at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?] at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?] at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?] at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?] at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?] at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?] at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?] at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?] at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?] at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?] at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?] at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$35(Security.java:1758) ~[?:?] at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$38(Security.java:1815) ~[?:?] at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?] at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:139) ~[?:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:356) ~[elasticsearch-8.13.4.jar:?] at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:113) ~[?:?] at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:1583) ~[?:?] ``` ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 18, 2024, 8:58am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/4 "2024-07-18T08:58:24Z")
> [@Mireiabm](#): > > `org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable` That doesn't correspond with > [@Mireiabm](#): > > `{"cluster_name":"prod","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":179,"active_shards":179,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}u` Are those messages from roughly the same time on the same cluster? If one of the security shards is unavailable then the cluster will not be green. ---
**Author:** ![Mireiabm](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@Mireiabm](https://discuss.elastic.co/u/Mireiabm)\ **Post date:** [July 18, 2024, 9:13am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/5 "2024-07-18T09:13:22Z")
**As you requested, the full message log is from today, and the ones below that I will attach are from right now.** ```auto Enter host password for user 'elastic': {"cluster_name":"prod","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":178,"active_shards":178,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}ubuntu@ip-10-100-100-11:~$ sudo curl --cacert /etsudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -X GET "https://localhost:9200/_cat/shards/.security-7?v" ``` ```auto index shard prirep state docs store dataset ip node .security-7 0 p STARTED 190 489.8kb 489.8kb 10.100.100.11 prod-1 ``` **I have included the two logs again for you to see that they are from right now.** ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 19, 2024, 2:41am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/6 "2024-07-19T02:41:28Z")
That implies that your cluster is now healthy. Have you tried to restart Kibana? Is it still broken? ---
**Author:** ![Mireiabm](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@Mireiabm](https://discuss.elastic.co/u/Mireiabm)\ **Post date:** [July 19, 2024, 7:24am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/7 "2024-07-19T07:24:58Z")
**I will take screenshots so you can see them yourself.** **I can't include two screenshots, so I'll provide one showing the code for restarting the services, the cluster status, and a command displaying the date to avoid any doubt.** ```auto ubuntu@ip:~$ sudo systemctl restart elasticsearch ubuntu@ip:~$ sudo systemctl restart kibana ubuntu@ip~$ sudo systemctl restart elasticsearch ubuntu@ip:~$ sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -X GET "https://localhost:9200/_cluster/health" Enter host password for user 'elastic': {"cluster_name":"comexicloud-prod","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":178,"active_shards":178,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}ubuntu@ip~$ sudo curl --cacert /etsudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -X GET "https://localhost:9200/_cat/shards/.security-7?v" Enter host password for user 'elastic': index shard prirep state docs store dataset ip node .security-7 0 p STARTED 190 489.8kb 489.8kb 10.100.100.11 comexicloud-prod-1 ubuntu@ip~$ date Fri Jul 19 07:14:16 UTC 2024 ``` And here is a screenshot from another terminal while I was restarting the Elasticsearch and Kibana services. ![elastic2](https://us1.discourse-cdn.com/elastic/original/3X/9/7/97c5536825a2fc086f3e89e6424e9e5f79f0cee1.png) Thank you ---
**Author:** ![Mireiabm](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@Mireiabm](https://discuss.elastic.co/u/Mireiabm)\ **Post date:** [July 19, 2024, 7:37am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/8 "2024-07-19T07:37:50Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b4a70db8308af51ecf13f11eba0ec31d25629a88.png) ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 19, 2024, 8:26am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/9 "2024-07-19T08:26:20Z")
If you restart Elasticsearch and Kibana at the same time then you will have errors. Elasticsearch takes some time to restart and Kibana will be unable to connect while Elasticsearch is starting. Is Kibana working now? If not, what error does it show in the logs? If it's not working, try restarting _just Kibana_. ---
**Author:** ![Mireiabm](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@Mireiabm](https://discuss.elastic.co/u/Mireiabm)\ **Post date:** [July 19, 2024, 8:41am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/10 "2024-07-19T08:41:15Z")
**I have restarted only Kibana, and I am still getting the message "Elastic did not load properly."** **These are the Kibana logs after restarting.** **It seems strange because I have always used the free tools, and this happened overnight without changing anything at all.** ```auto ubuntu@ip:~$ sudo tail -f /var/log/kibana/kibana.log | grep ERROR {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-07-19T08:38:10.577+00:00","message":"Failed to resolve ELSER model definition: Error: Platinum, Enterprise or trial license needed","log":{"level":"ERROR","logger":"plugins.observabilityAIAssistant"},"process":{"pid":26387,"uptime":34.805928092},"trace":{"id":"033236d44a915e89b90ff283aff4759a"},"transaction":{"id":"104098f0d26ddd0c"}} ``` ```auto ubuntu@ip-:~$ sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -X GET "https://localhost:9200/_license" Enter host password for user 'elastic': { "license" : { "status" : "active", "uid" : "38b80e22-5013-4157-880b-9f7ecc96c2f1", "type" : "basic", "issue_date" : "2023-06-11T21:45:01.719Z", "issue_date_in_millis" : 1686519901719, "max_nodes" : 1000, "max_resource_units" : null, "issued_to" : "elasticsearch", "issuer" : "elasticsearch", "start_date_in_millis" : -1 } } ``` ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 22, 2024, 1:55am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/11 "2024-07-22T01:55:49Z")
I'm going to move this to the Kibana forum. There's nothing here to indicate an Elasticsearch problem, and the Kibana community are more likely to have answers about why Kibana isn't loading. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 22, 2024, 1:56am UTC](https://discuss.elastic.co/t/failed-to-retrieve-password-hash-for-reserved-user-kibana-system/363271/12 "2024-07-22T01:56:03Z")
From #Elasticsearch to #Kibana