Kibana unable to authenticate

Hello,

I have been running into an issue with my Kibana dashboard. Currently, it shows "Kibana server is not ready yet". When I use the journalctl -xe command, I see the following error for Kibana:

["warning","plugins","licensing"],"pid":6937,"message":"License information could not be obtained from Elasticsearch due to [security_exception] unable to authenticate user [kibana] for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }

It sounds like there is some sort of authentication error. I had setup the elasticsearch-setup-passwords tool before and everything had been working correctly before. But since I had setup the passwords for the elastic users prior to this issue, I am unable to reset those passwords. I tried using the elasticsearch-users passwd function and get the following error:

ERROR: Invalid username [elastic]... Username [elastic] is reserved and may not be used.

I get this error regardless of user (elastic, kibana, etc). I should note that this issue started occuring when I removed all prior indices to free space. The partition I had for the indices filled up very quickly and caused Elasticsearch and Kibana to crash. I should also note that I am unable to authenticate using cURL

curl -u elastic'http://localhost:9200/_xpack/security/_authenticate?pretty'

Result:
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "unable to authenticate user [elastic] for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "unable to authenticate user [elastic] for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

Is there anyway I can fix this error? Would I have to re-setup the passwords for the elastic users? I am running Elasticsearch Version 7.9.2

Thank you for any and all help

Have you tried logging into Elasticsearch directly, with curl?
Have you checked for more information in the Elasticsearch logs?

When you removed indices to clear space, were any system indices removed? I just want to rule out a possible accidental removal of the security index: you should have an index called .security-7

I am also having similar issue. not able to authenticate Kibana after installing kibana with searchguard. using basicauth and logging in with default user& pw as "kibanaserver". Any suggestions please.

"name":"Error","stack":"Error: 140254743959424:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\

Please create your own topic for your question :slight_smile:

Hey Tim,

Here is the error within the elasticsearch.log:

org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:181) ~[x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:525) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:225) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:99) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:167) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:104) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:448) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) [x-pack-core-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:503) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:415) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:425) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:366) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:345) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:347) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:329) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:405) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:325) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:141) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:126) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:63) [x-pack-security-7.9.2.jar:7.9.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:318) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:308) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-7.9.2.jar:7.9.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) [transport-netty4-client-7.9.2.jar:7.9.2]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.9.2.jar:7.9.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]

It is my understanding that we removed all indices, including the security index.

Thanks,

Hi Daniel, unfortunately, it looks like Elastic Security is not going to work until you restore the .security-7 index. Did you create backups of the data before removing indices?

Since this is more of an Elasticsearch operations issue, you may have better luck talking to the experts in the Elasticsearch category: https://discuss.elastic.co/c/elastic-stack/elasticsearch/6

The _cat/allocation and _cat/shards APIs are going to be your friend:

  • https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html
  • https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-allocation.html

Hey Tim,

Thank you so much for helping me understand this issue. It looks like we did not take back ups of those indices. Foolish, I know. Is there any way I can recreate a new .security-7 index? If I can, I would like to try to avoid re-installing ElasticSearch, but if I have to then so be it.

Again, thank you so much for the assistance.

Best,

Based on a thread I found in the Elasticsearch category, you can restart the cluster set the user passwords for built-in users. See Accidentally deleted .security index for x-pack

Hey Tim,

Thanks for showing me that thread. I am still have issues trying to reset the kibana password however. I tried to restart my cluster based on this doc I found:

I'm not sure if I did something wrong here, when I try to do a synced-flush, I get these results:

".kibana-event-log-7.9.2-000001" : {
"total" : 1,
"successful" : 0,
"failed" : 1,
"failures" : [
{
"shard" : 0,
"reason" : "no active shards"
}
]
},
"wazuh-monitoring-3.x-2020.10.10" : {
"total" : 2,
"successful" : 0,
"failed" : 2,
"failures" : [
{
"shard" : 0,
"reason" : "no active shards"
},
{
"shard" : 1,
"reason" : "no active shards"
}
]
},
".kibana-event-log-7.9.2-000004" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
".kibana-event-log-7.9.2-000002" : {
"total" : 1,
"successful" : 0,
"failed" : 1,
"failures" : [
{
"shard" : 0,
"reason" : "no active shards"
}
]
},
".kibana-event-log-7.9.2-000003" : {
"total" : 1,
"successful" : 0,
"failed" : 1,
"failures" : [
{
"shard" : 0,
"reason" : "no active shards"
}
]
},

I follow the instructions from there and I am unable to reset the passwords per the instructions:

I created my own super user and tried to rest the elastic and kibana passwords via this curl command:

curl -u my_admin -XPUT 'http://localhost:9200/_xpack/security/user/elastic/_password?pretty' -H 'Content-Type: application/json' -d'
{
  "password" : "new_password"
}
' 

This returns the following results:

{
"error" : {
"root_cause" : [
{
"type" : "unavailable_shards_exception",
"reason" : "[.security-7][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"
}
],
"type" : "unavailable_shards_exception",
"reason" : "[.security-7][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"
},
"status" : 503
}

I even attempted to try to set a bootstrap password for elasticsearch and no luck there being able to authenticate:

bin/elasticsearch-keystore add "bootstrap.password"

{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "unable to authenticate user [elastic] for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "unable to authenticate user [elastic] for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

Any guidance here?

Thanks