Our Cerebro shows the cluster in green, all of our 9 nodes are there with ES: 8.3.2.
There are no [ERROR]s in Kibana's log, just [INFO ]s and [WARN ]s.
This appeared in the process of upgrading our dev cluster and we will not upgrade our prod cluster unless this works properly there. ATM we have 2,732 indices, 5,465 shards, 3,826,484,223 docs, 2.21 TB in our prod cluster, so we want to be absolutely sure that everything works before upgrading it.
I checked the log of the current master node (Is this the one that handles Kibana logins?). The only lines with "error" are:
Line 1: 2022-09-19 10:08:38,122 main INFO Cannot initialize scripting support because this JRE does not support it. java.lang.NoClassDefFoundError: javax/script/ScriptEngineManager
Line 291: at org.elasticsearch.indices.recovery.RecoverySourceHandler$OperationBatchSender.handleError(RecoverySourceHandler.java:1209) ~[elasticsearch-8.3.2.jar:?]
Line 292: at org.elasticsearch.indices.recovery.RecoverySourceHandler$OperationBatchSender.handleError(RecoverySourceHandler.java:1126) ~[elasticsearch-8.3.2.jar:?]
The msg on line 1 appears also in the logs of our other two master nodes.
W.R.T. the kibanaserver and cerebro errors in Kibana UI:
The role kibana_system is only for background administration on an Elasticsearch cluster. That role contains internal-only privileges, no user-level permissions to access the UI. You will need to use a different role for Kibana UI login.
The role admin sounds like it might be custom. Check if it has the appropriate privileges as mentioned in Kibana 8.3.2 docs. Privileges may have changed between 7.17.1 and 8.3.2.
Given the permission error you are having when logging into Kibana, it is likely Elasticsearch authentication/authorization system is working fine. To solve your original problem, you can give user cerebro the superuser role temporarily. Login with it and change the password for the elastic user. You can then remove revert the changes to user cerebro after it is done. To give user cerebro the superuser role, just add the following line to the users_roles file:
PS: Once you made the change, the ES node will automoatically reload the file in at most 5 seconds.