Kibana_system password reset but only affected one master node in the ELK cluster

Elastic Stack Kibana
1

Hi,

I'm using an ELK cluster with two master nodes elastic01 and elastic02. Cluster has been upgraded from version 7.17 to 8.2. I've reset kibana_system password on elastic01 but it didn't reset on second master node (elastic02). If I try to reset on elastic02:

/usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system -i

it returned:

ERROR: Failed to determine the health of the cluster. Unexpected http status [503]

On my kibana01 node, I've got this errors for kibana.services:

sep 03 18:00:16 qa1kibana01.registeredsite.com kibana[7288]: [2023-09-03T18:00:16.267-04:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
sep 03 18:00:16 qa1kibana01.registeredsite.com kibana[7288]: [2023-09-03T18:00:16.359-04:00][INFO ][status] Kibana is now critical (was degraded)
sep 03 18:00:18 qa1kibana01.registeredsite.com kibana[7288]: [2023-09-03T18:00:18.805-04:00][ERROR][plugins.security.session.index] Failed to check if session legacy index template exists:
sep 03 18:00:18 qa1kibana01.registeredsite.com kibana[7288]: [2023-09-03T18:00:18.806-04:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_security/privilege/kibana-.kibana]

Cluster configuration at kibana.yml:

elasticsearch.hosts: ["http://elastic01.registeredsite.com:9200", "http://elastic02.registeredsite.com:9200"]

Cluster configuration at elasticsearch.yml on elastic01 and elastic02:

discovery.seed_hosts: ["elastic01", "elastic02"]

Could anyone help me? I don't know what else to check.
Thanks in advance!

2

Any reason to reset the password? There is no need to reset it just because of the upgrade.

Also, the users and passwords are stored in an index, so every node in the cluster will have access to this indice, if you are reseted it, there is no need to do it again, if you do it, it will just reset the password in the cluster again.

This normally means that your cluster is not running, is the elasticsearch service running in both nodes and your cluster is working? What do you have in the log of both your nodes?

Did you update the password in kibana.yml and restarted it? This means that Kibana can not authenticate itself in the cluster.

This may also be an issue, with just two nodes you should set just one as the master, so you should choose one of the nodes to be the master and remove the master role from the other one.

3 
Any reason to reset the password? There is no need to reset it just because of the upgrade.

I updated because I wasn't sure my password was correct. Now if I validate it on node elastic01 it works but fails on node elastic02.

This normally means that your cluster is not running, is the elasticsearch service running in both nodes and your cluster is working?

elasticsearch service is running on both nodes

Did you update the password in kibana.yml and restarted it? This means that Kibana can not authenticate itself in the cluster.

Yes, I updated the password in kibana.yml and restarted kibana service. But kibana fails to authenticate against node elastic02.

This may also be an issue, with just two nodes you should set just one as the master, so you should choose one of the nodes to be the master and remove the master role from the other one.

Fixed!

 What do you have in the log of both your nodes?

These are logs from elastic02 after start service:

[2023-09-04T11:27:11,666][INFO ][o.e.n.Node               ] [qa1elastic02] initialized
[2023-09-04T11:27:11,666][INFO ][o.e.n.Node               ] [qa1elastic02] starting ...
[2023-09-04T11:27:11,688][INFO ][o.e.x.s.c.f.PersistentCache] [qa1elastic02] persistent cache index loaded
[2023-09-04T11:27:11,689][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [qa1elastic02] deprecation component started
[2023-09-04T11:27:11,831][INFO ][o.e.t.TransportService   ] [qa1elastic02] publish_address {qa1elastic02/10.201.70.104:9300}, bound_addresses {10.201.70.104:9300}
[2023-09-04T11:27:14,077][INFO ][o.e.b.BootstrapChecks    ] [qa1elastic02] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-09-04T11:27:14,098][INFO ][o.e.c.c.Coordinator      ] [qa1elastic02] cluster UUID [9XTxFStHQhOe40BbR7nBAQ]
[2023-09-04T11:27:24,111][WARN ][o.e.c.c.ClusterFormationFailureHelper] [qa1elastic02] master not discovered or elected yet, an election requires a node with id [GIgM4Gu4QCiBypu3KeRMVQ], have only discovered non-quorum [{qa1elastic02}{8_uhebapTRyINP_vZcrP9Q}{OdTqTqFcQlaWZfSMk0pnqg}{qa1elastic02}{10.201.70.104:9300}{cdfhilmrstw}]; discovery will continue using [10.201.70.103:9300] from hosts providers and [{qa1elastic02}{8_uhebapTRyINP_vZcrP9Q}{OdTqTqFcQlaWZfSMk0pnqg}{qa1elastic02}{10.201.70.104:9300}{cdfhilmrstw}] from last-known cluster state; node term 237, last-accepted version 77399 in term 228
[2023-09-04T11:27:34,113][WARN ][o.e.c.c.ClusterFormationFailureHelper] [qa1elastic02] master not discovered or elected yet, an election requires a node with id [GIgM4Gu4QCiBypu3KeRMVQ], have only discovered non-quorum [{qa1elastic02}{8_uhebapTRyINP_vZcrP9Q}{OdTqTqFcQlaWZfSMk0pnqg}{qa1elastic02}{10.201.70.104:9300}{cdfhilmrstw}]; discovery will continue using [10.201.70.103:9300] from hosts providers and [{qa1elastic02}{8_uhebapTRyINP_vZcrP9Q}{OdTqTqFcQlaWZfSMk0pnqg}{qa1elastic02}{10.201.70.104:9300}{cdfhilmrstw}] from last-known cluster state; node term 237, last-accepted version 77399 in term 228
[2023-09-04T11:27:44,110][WARN ][o.e.n.Node               ] [qa1elastic02] timed out while waiting for initial discovery state - timeout: 30s
[2023-09-04T11:27:44,115][WARN ][o.e.c.c.ClusterFormationFailureHelper] [qa1elastic02] master not discovered or elected yet, an election requires a node with id [GIgM4Gu4QCiBypu3KeRMVQ], have only discovered non-quorum [{qa1elastic02}{8_uhebapTRyINP_vZcrP9Q}{OdTqTqFcQlaWZfSMk0pnqg}{qa1elastic02}{10.201.70.104:9300}{cdfhilmrstw}]; discovery will continue using [10.201.70.103:9300] from hosts providers and [{qa1elastic02}{8_uhebapTRyINP_vZcrP9Q}{OdTqTqFcQlaWZfSMk0pnqg}{qa1elastic02}{10.201.70.104:9300}{cdfhilmrstw}] from last-known cluster state; node term 237, last-accepted version 77399 in term 228
[2023-09-04T11:27:44,117][INFO ][o.e.h.AbstractHttpServerTransport] [qa1elastic02] publish_address {qa1elastic02/10.201.70.104:9200}, bound_addresses {10.201.70.104:9200}
[2023-09-04T11:27:44,118][INFO ][o.e.n.Node               ] [qa1elastic02] started
[2023-09-04T11:27:50,842][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:50,846][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:50,846][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:50,848][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:50,848][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:50,849][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:50,849][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:50,851][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:50,962][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:51,308][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-09-04T11:27:54,116][WARN ][o.e.c.c.ClusterFormationFailureHelper] [qa1elastic02] master not discovered or elected yet, an election requires a node with id [GIgM4Gu4QCiBypu3KeRMVQ], have only discovered non-quorum [{qa1elastic02}{8_uhebapTRyINP_vZcrP9Q}{OdTqTqFcQlaWZfSMk0pnqg}{qa1elastic02}{10.201.70.104:9300}{cdfhilmrstw}]; discovery will continue using [10.201.70.103:9300] from hosts providers and [{qa1elastic02}{8_uhebapTRyINP_vZcrP9Q}{OdTqTqFcQlaWZfSMk0pnqg}{qa1elastic02}{10.201.70.104:9300}{cdfhilmrstw}] from last-known cluster state; node term 237, last-accepted version 77399 in term 228
[2023-09-04T11:27:54,933][INFO ][o.e.x.s.a.RealmsAuthenticator] [qa1elastic02] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

and these logs from master elastic01:

[2023-09-04T11:16:59,992][INFO ][o.e.n.Node               ] [qa1elastic01] node name [qa1elastic01], node ID [GIgM4Gu4QCiBypu3KeRMVQ], cluster name [qa-elk-cluster], roles [ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen]
[2023-09-04T11:17:04,608][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [qa1elastic01] [controller/24395] [Main.cc@123] controller (64 bit): Version 8.2.3 (Build 537f37a54d22f1) Copyright (c) 2022 Elasticsearch BV
[2023-09-04T11:17:04,773][INFO ][o.e.x.s.Security         ] [qa1elastic01] Security is enabled
[2023-09-04T11:17:05,162][INFO ][o.e.x.s.a.Realms         ] [qa1elastic01] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-09-04T11:17:05,169][INFO ][o.e.x.s.a.s.FileRolesStore] [qa1elastic01] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-09-04T11:17:06,638][INFO ][o.e.t.n.NettyAllocator   ] [qa1elastic01] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2023-09-04T11:17:06,667][INFO ][o.e.i.r.RecoverySettings ] [qa1elastic01] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-09-04T11:17:06,699][INFO ][o.e.d.DiscoveryModule    ] [qa1elastic01] using discovery type [multi-node] and seed hosts providers [settings]
[2023-09-04T11:17:07,830][INFO ][o.e.n.Node               ] [qa1elastic01] initialized
[2023-09-04T11:17:07,831][INFO ][o.e.n.Node               ] [qa1elastic01] starting ...
[2023-09-04T11:17:07,855][INFO ][o.e.x.s.c.f.PersistentCache] [qa1elastic01] persistent cache index loaded
[2023-09-04T11:17:07,856][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [qa1elastic01] deprecation component started
[2023-09-04T11:17:07,968][INFO ][o.e.t.TransportService   ] [qa1elastic01] publish_address {qa1elastic01/10.201.70.103:9300}, bound_addresses {10.201.70.103:9300}
[2023-09-04T11:17:10,855][INFO ][o.e.b.BootstrapChecks    ] [qa1elastic01] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-09-04T11:17:10,875][INFO ][o.e.c.c.Coordinator      ] [qa1elastic01] cluster UUID [9XTxFStHQhOe40BbR7nBAQ]
[2023-09-04T11:17:11,037][INFO ][o.e.c.s.MasterService    ] [qa1elastic01] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {qa1elastic01}{GIgM4Gu4QCiBypu3KeRMVQ}{v9UdveF_TzW3SiBCwTWAEw}{qa1elastic01}{10.201.70.103:9300}{cdfhilmrstw} completing election], term: 238, version: 78568, delta: master node changed {previous [], current [{qa1elastic01}{GIgM4Gu4QCiBypu3KeRMVQ}{v9UdveF_TzW3SiBCwTWAEw}{qa1elastic01}{10.201.70.103:9300}{cdfhilmrstw}]}
[2023-09-04T11:17:11,240][INFO ][o.e.c.s.ClusterApplierService] [qa1elastic01] master node changed {previous [], current [{qa1elastic01}{GIgM4Gu4QCiBypu3KeRMVQ}{v9UdveF_TzW3SiBCwTWAEw}{qa1elastic01}{10.201.70.103:9300}{cdfhilmrstw}]}, term: 238, version: 78568, reason: Publication{term=238, version=78568}
[2023-09-04T11:17:11,272][INFO ][o.e.h.AbstractHttpServerTransport] [qa1elastic01] publish_address {qa1elastic01/10.201.70.103:9200}, bound_addresses {10.201.70.103:9200}
[2023-09-04T11:17:11,273][INFO ][o.e.n.Node               ] [qa1elastic01] started
[2023-09-04T11:17:11,301][INFO ][o.e.c.s.ClusterSettings  ] [qa1elastic01] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2023-09-04T11:17:11,900][INFO ][o.e.l.LicenseService     ] [qa1elastic01] license [5a636afe-d560-41c9-96da-c62fadd74734] mode [basic] - valid
[2023-09-04T11:17:11,901][INFO ][o.e.x.s.a.Realms         ] [qa1elastic01] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-09-04T11:17:11,904][INFO ][o.e.g.GatewayService     ] [qa1elastic01] recovered [81] indices into cluster_state
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.