Kibana login does not work

I have enabled inter-node TLS communication and xpack security as described here:


I have set passwords for elastic, kibana and all other users with

Blockquote
elasticsearch-setup-passwords interactive
Blockquote

i have checked with user elastic several API endpoints (_cluster, nodes, shards etc..) and i got proper response.
i run kibana app in docker on each elasticsearch node ( master and datanode) and I have load balancer to distribute requests between 3 these nodes. I have enabled xpack security in kibana.yml either.
Right now when i try o access URL of load balancer I see normal kibana login prompt, but it does not let me in. I tried to login with user elastic and kibana. I don`t get any error message, just login prompt again. Have to notice: if i use intentionally incorrect password for user 'elastic' i receive error message about 'incorrect user/password', but when i use the correct one I just observe kibana logo moving around itself several seconds like it doing something, but then i return to the login prompt again. No error in kibana log.
I see in elasticearch log the following errors(?):
Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

elasticsearch and kibana versions: 7.1.1
kibana config:

Blockquote
server.name: es-master-3
server.host: 0.0.0.0
xpack.security.enabled: true
elasticsearch.hosts: "http://127.0.0.1:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
xpack.license_management.enabled: True
xpack.monitoring.ui.container.elasticsearch.enabled: True
Blockquote

Please give me some directions.

Update: I have enabled verbose logging on kibana and this is what i get:

Blockquote
log [19:29:50.720] [debug][plugin] Checking Elasticsearch version
log [19:29:51.713] [debug][legacy-proxy] Event is being forwarded: connection
log [19:29:51.713] [debug][legacy-service] Request will be handled by proxy GET:/app/kibana.
error [19:29:51.714] [debug][auth][security][session] Error: Unauthorized
at validate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:153:49)
at Object.authenticate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:226:26)
at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)
at module.exports.internals.Auth.test (/usr/share/kibana/node_modules/hapi/lib/auth.js:92:54)
at Session.get (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/session.js:56:47)
at Authenticator.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:132:49)
at Object.server.expose.request [as authenticate] (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:288:60)
at Object.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/auth_redirect.js:28:60)
at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)
at module.exports.internals.Auth._authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:238:58)
at authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:214:21)
at Request._lifecycle (/usr/share/kibana/node_modules/hapi/lib/request.js:263:62)
at process._tickCallback (internal/process/next_tick.js:68:7)

I assume this is somehow related to fact that kibana runs in docker container, and elasticsearch runs as application on the server (installed from deb package)

User/password for kibana user seems to be correct
root@es-master-1:/srv/kibana# curl -u kibana 'http://localhost:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'kibana':
{
"username" : "kibana",
"roles" : [
"kibana_system"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true,
"authentication_realm" : {
"name" : "reserved",
"type" : "reserved"
},
"lookup_realm" : {
"name" : "reserved",
"type" : "reserved"
}
}

Hi @dumkaz,

Make sure you have xpack.security.encryptionKey set correctly in your kibana.yml.
This must be set to the same value across all kibana instances in order for a load-balanced installation to work correctly

@Larry_Gregory
where can i read about xpack.security.encryptionKey and how to use it for load balancing ?
Actually the problem was AWS ALB. When connected directly to the instance IP as
https://hostname:5601 all worked as a charm. I also noticed in tcpdump issues with several HTTP headers on AWS LB ( like X-Forwarded-For )

@dumkaz,

The docs don't do a great job of spelling this out for load-balancing, but the description of the key is outlined here: https://www.elastic.co/guide/en/kibana/current/security-settings-kb.html

This key is what Kibana uses to encrypt the user's session cookie. If you don't configure one, then Kibana generates a random key. Since each instance would generate its own random key by default, that means that each instance won't be able to read the cookies generated by the other instances. By setting this value the same across all instances, you're allowing then to read the cookie value, no matter which instance generated the cookie.

Oh, this is fantastic. This solves now the problem with using load balancer. Thank you very much

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.