Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]

[arb (Armand)]

I am having a problem with a clean installation of elastic & kibana (Basic).
Steps so far:

1. install Elasticsearch (7.4.1)

2. install kibana (7.4.1)
   each works fine and can communicate to each other.

3. Add the configuration for security on elastic
   xpack.security.enabled: true

4. Generate passwords through
   bin/elasticsearch-setup-passwords -auto

5. Authenticate through elasticsearch :9200 works fine

6. Configured kibana.yml to include
   elasticsearch.username: "kibana"
   elasticsearch.password: "whatever"

7. Restart kibana

Kibana opens, shows a login screen.
Although password is correct, it is not possible to authenticate. Only log is:
Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]

I checked the indexes in ES:
green open .security-7 Pk6wepWdQ4a2nr1IqcremQ 1 0 42 0 64.1kb 64.1kb
green open .kibana_task_manager_1 s1VvONQySYWT3qYkt7lMpQ 1 0 2 0 46kb 46kb
green open .apm-agent-configuration Id8vVfukTNuSWJ7LFwi0OA 1 0 0 0 283b 283b
green open .kibana_1 6mwaNEFzTNWlcR1IPRvERA 1 0 2 0 11.2kb 11.2kb
I checked these other threads, that were closed due to passing 28 days:

Can't Authenticate Kibana to Elasticsearch

Running ELK version 6.0 on Ubuntu 16.0.4 LTS virtual machine. I have run through the X-Pack setup guide and created the passwords for: elastic, kibana and logstash_system. I then added the kibana username and password to the kibana.yml file under elasticsearch.username and elasticsearch.password, I then restarted ES and Kibana. Now when I go to the Kibana web interface I am not asked for login (I thought X-Pack would?). I see this in Kibana: Elasticsearch plugin is red Unable to connect to …

Elasticsearch start log error “failed to authenticate user [kibana]”

My elasticsearch has been reporting the following error during startup, I hope professional personnel help solve it, thank you very much! [2018-05-11T09:24:34,959][INFO ][o.e.x.s.a.AuthenticationService] [elk-node2] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana] [2018-05-11T09:24:37,555][INFO ][o.e.x.s.a.AuthenticationService] [elk-node2] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana] [20…

Authentication of [kibana] was terminated by realm [reserved]

Hi, I converted a elasticsearch (6.3) standalone to a new cluster, after that kibana user is not authenticated and none of the reserved users are getting authenticated. And also i am not able to change the previously set passwords for the users when it was standalone. Also, getting license information could not be obtained from elasticsearch for the cluster. [elk-master-node] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana] {"type":"log","@t…

Elasticsearch ERROR: Failed to verify bootstrap password ---

Hi after ii upgraded to platinum license and then reset the password in elasticsearch 7.0 (bin/elasticsearch-setup-passwords interactive). i could able to login in elastic user. but i couldnt able to login using kibana user. In browser shows "{"statusCode":403,"error":"Forbidden","message":"Forbidden"}" and i tried to reset the password again in elasticsearch node. shows below error: [root@elastic1 elasticsearch]# bin/elasticsearch-setup-passwords interactive Failed to authenticate user …

What am I missing in the configurations?

[markov00 (Marco Vettorello)]

Hi @arb,

Can you please share the logs from ES that precede and follow the authentication error?
I just want to double check if there is something else that we are missing.

could you also provide me the method of installation for es and kibana and the current machine configuration?
thanks

[arb (Armand)]

Thank you for following up

@markov00

Can you please share the logs from ES that precede and follow the authentication error?
I just want to double check if there is something else that we are missing.

      [node-1] loaded module [ingest-common]

[2019-10-28T13:39:21,200][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-analytics]
[2019-10-28T13:39:21,201][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ccr]
[2019-10-28T13:39:21,201][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-core]
[2019-10-28T13:39:21,201][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-deprecation]
[2019-10-28T13:39:21,201][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-graph]
[2019-10-28T13:39:21,202][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ilm]
[2019-10-28T13:39:21,202][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-logstash]
[2019-10-28T13:39:21,202][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ml]
[2019-10-28T13:39:21,202][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-monitoring]
[2019-10-28T13:39:21,202][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-rollup]
[2019-10-28T13:39:21,203][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-security]
[2019-10-28T13:39:21,203][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-sql]
[2019-10-28T13:39:21,203][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-voting-only-node]
[2019-10-28T13:39:21,203][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-watcher]
[2019-10-28T13:39:21,204][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2019-10-28T13:39:24,607][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/opt/elasticsearch-7.4.1/config/roles.yml]
[2019-10-28T13:39:25,051][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/3531] [Main.cc@110] controller (64 bit): Version 7.4.1 (Build 973380bdacc5e8) Copyright (c) 2019 Elasticsearch BV
[2019-10-28T13:39:25,448][DEBUG][o.e.a.ActionModule       ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-10-28T13:39:25,865][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2019-10-28T13:39:26,472][INFO ][o.e.n.Node               ] [node-1] initialized
[2019-10-28T13:39:26,473][INFO ][o.e.n.Node               ] [node-1] starting ...
[2019-10-28T13:39:26,564][INFO ][o.e.t.TransportService   ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2019-10-28T13:39:26,610][WARN ][o.e.b.BootstrapChecks    ] [node-1] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-10-28T13:39:26,611][WARN ][o.e.b.BootstrapChecks    ] [node-1] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-10-28T13:39:26,611][WARN ][o.e.b.BootstrapChecks    ] [node-1] Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
[2019-10-28T13:39:26,615][INFO ][o.e.c.c.Coordinator      ] [node-1] cluster UUID [LMxW-5sdQUmmSxu1f8vliw]
[2019-10-28T13:39:26,624][INFO ][o.e.c.c.ClusterBootstrapService] [node-1] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2019-10-28T13:39:26,880][INFO ][o.e.c.s.MasterService    ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{m3WFvelRRL2Vhx63ywYCOg}{hJaKDNWJS8mCF78SoCLVgQ}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=33649274880, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 37, reason: master node changed {previous [], current [{node-1}{m3WFvelRRL2Vhx63ywYCOg}{hJaKDNWJS8mCF78SoCLVgQ}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=33649274880, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-10-28T13:39:27,327][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-1}{m3WFvelRRL2Vhx63ywYCOg}{hJaKDNWJS8mCF78SoCLVgQ}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=33649274880, xpack.installed=true, ml.max_open_jobs=20}]}, term: 3, version: 37, reason: Publication{term=3, version=37}
[2019-10-28T13:39:27,469][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {PUBLICIP:9200}, bound_addresses {127.0.0.1:9200}, {PUBLICIP:9200}
[2019-10-28T13:39:27,470][INFO ][o.e.n.Node               ] [node-1] started
[2019-10-28T13:39:28,046][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
....
[2019-10-28T13:39:28,114][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
[2019-10-28T13:39:28,179][INFO ][o.e.l.LicenseService     ] [node-1] license [bf0cb628-24ec-4f0c-8d3d-f394c926bf05] mode [basic] - valid
[2019-10-28T13:39:28,180][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is enabled
[2019-10-28T13:39:28,187][INFO ][o.e.g.GatewayService     ] [node-1] recovered [4] indices into cluster_state
[2019-10-28T13:39:29,633][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.security-7][0], [.kibana_task_manager_1][0], [.kibana_1][0]]]).
[2019-10-28T13:39:30,682][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.management-beats] for index patterns [.management-beats]

@markov00

could you also provide me the method of installation for es and kibana and the current machine configuration?

Both are installed by deploying the .tar.gz files.
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
openjdk version "1.8.0_232"

    cluster.name: myclustername

    node.name: node-1

    path.data: /home/myclustername/data-elastic

    path.logs: /home/myclustername/logs-elastic

    network.host: 127.0.0.1
    http.host: ["PUBLICIP","127.0.0.1"]

    http.port: 9200
    xpack.security.enabled: true

[markov00 (Marco Vettorello)]

Thanks for posting this.
I've tested it locally and seems to be a reproducible issue in macOS also.
BTW I successfully logged in using a elastic as user

[markov00 (Marco Vettorello)]

I will contact the security team about that and open an issue in Kibana GH repo if this is an unwanted behaviour

[arb (Armand)]

@markov00

the security team about that and open an issue in Kibana GH repo if this is an unwanted behaviour

I cannot login even with user elastic.

[markov00 (Marco Vettorello)]

I've spoke with the security team. They stated that you cannot login with the kibana user created by the bin/elasticsearch-setup-passwords because it's a system role and it's used internally by kibana itself and can't be used externally, here you can find a GH ticket were we will deprecate that name: https://github.com/elastic/kibana/issues/25879

You should be able to login with the elastic user and create any other required Kibana users from the UI.
If you can login with the elastic user, try to restart your browser, this should clear the cache and allow you to enter the login credentials.

[arb (Armand)]

Thank you for bearing with me.
Yes I confirmed in a new browser I was able to login with elastic (strange it would not on the original browser).

I was able to follow the instructions on https://www.elastic.co/guide/en/elasticsearch/reference/7.4/get-started-users.html to create further users in Kibana