Failed to retrieve password hash for reserved user kibana_system

Elastic Stack Kibana
1

hi,
From one day to the next, my Elasticsearch stopped working without me doing anything, and this error appears: [2024-07-11T09:22:37,457][ERROR][o.e.x.s.a.e.ReservedRealm] [prod-1] failed to retrieve password hash for reserved user [kibana_system]

What I did was change the password for kibana_system, update it in the kibana.yml file, restart Elasticsearch, and it still doesn't work. The cluster status appears correct, and the password verification for kibana_system also checks out fine. I don't know what else I can do.

They know what might be happening; everything seems fine.


Enter host password for user 'elastic':
{"cluster_name":"prod","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":179,"active_shards":179,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}u

index       shard prirep state   docs   store dataset ip            node
.security-7 0     p      STARTED  190 489.8kb 489.8kb Ip     prod-1

I have changed and configured the new password for kibana_system, the cluster is green, all shards are started. What else can I look at?

Elasticsearch Version
8.14.2

ubuntu 22.04

Thanks a lot!

1 Like
2

Is there more to that error message? It is highly unlikely for the logs to print that message without additional details.

3 
[2024-07-18T06:56:20,286][ERROR][o.e.x.s.a.e.ReservedRealm] [prod-1] failed to retrieve password hash for reserved user [kibana_system]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
        at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:178) ~[?:?]
        at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:640) ~[?:?]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:272) ~[?:?]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:136) ~[?:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) ~[?:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) ~[?:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:263) ~[?:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:146) ~[?:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:95) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:130) ~[?:?]
        at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$35(Security.java:1758) ~[?:?]
        at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$38(Security.java:1815) ~[?:?]
        at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:47) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:139) ~[?:?]
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:356) ~[elasticsearch-8.13.4.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:113) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
4

That doesn't correspond with

Are those messages from roughly the same time on the same cluster?
If one of the security shards is unavailable then the cluster will not be green.

5

As you requested, the full message log is from today, and the ones below that I will attach are from right now.

Enter host password for user 'elastic':
{"cluster_name":"prod","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":178,"active_shards":178,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}ubuntu@ip-10-100-100-11:~$ sudo curl --cacert /etsudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -X GET "https://localhost:9200/_cat/shards/.security-7?v"


index       shard prirep state   docs   store dataset ip            node
.security-7 0     p      STARTED  190 489.8kb 489.8kb 10.100.100.11 prod-1

I have included the two logs again for you to see that they are from right now.

6

That implies that your cluster is now healthy.
Have you tried to restart Kibana? Is it still broken?

7

I will take screenshots so you can see them yourself.

I can't include two screenshots, so I'll provide one showing the code for restarting the services, the cluster status, and a command displaying the date to avoid any doubt.

ubuntu@ip:~$ sudo systemctl restart elasticsearch
ubuntu@ip:~$ sudo systemctl restart kibana
ubuntu@ip~$ sudo systemctl restart elasticsearch
ubuntu@ip:~$ sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -X GET "https://localhost:9200/_cluster/health"
Enter host password for user 'elastic':
{"cluster_name":"comexicloud-prod","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":178,"active_shards":178,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}ubuntu@ip~$ sudo curl --cacert /etsudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -X GET "https://localhost:9200/_cat/shards/.security-7?v"
Enter host password for user 'elastic':
index       shard prirep state   docs   store dataset ip            node
.security-7 0     p      STARTED  190 489.8kb 489.8kb 10.100.100.11 comexicloud-prod-1
ubuntu@ip~$ date
Fri Jul 19 07:14:16 UTC 2024

And here is a screenshot from another terminal while I was restarting the Elasticsearch and Kibana services.

elastic2
elastic21895×628 111 KB

Thank you

8

image
image1446×210 11.6 KB

9

If you restart Elasticsearch and Kibana at the same time then you will have errors. Elasticsearch takes some time to restart and Kibana will be unable to connect while Elasticsearch is starting.

Is Kibana working now? If not, what error does it show in the logs?

If it's not working, try restarting just Kibana.

10

I have restarted only Kibana, and I am still getting the message "Elastic did not load properly."

These are the Kibana logs after restarting.
It seems strange because I have always used the free tools, and this happened overnight without changing anything at all.

ubuntu@ip:~$ sudo tail -f /var/log/kibana/kibana.log | grep ERROR
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-07-19T08:38:10.577+00:00","message":"Failed to resolve ELSER model definition: Error: Platinum, Enterprise or trial license needed","log":{"level":"ERROR","logger":"plugins.observabilityAIAssistant"},"process":{"pid":26387,"uptime":34.805928092},"trace":{"id":"033236d44a915e89b90ff283aff4759a"},"transaction":{"id":"104098f0d26ddd0c"}}

ubuntu@ip-:~$ sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -X GET "https://localhost:9200/_license"
Enter host password for user 'elastic':
{
  "license" : {
    "status" : "active",
    "uid" : "38b80e22-5013-4157-880b-9f7ecc96c2f1",
    "type" : "basic",
    "issue_date" : "2023-06-11T21:45:01.719Z",
    "issue_date_in_millis" : 1686519901719,
    "max_nodes" : 1000,
    "max_resource_units" : null,
    "issued_to" : "elasticsearch",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}
11

I'm going to move this to the Kibana forum. There's nothing here to indicate an Elasticsearch problem, and the Kibana community are more likely to have answers about why Kibana isn't loading.

1 Like
12

From Elasticsearch to Kibana