No login with 'elastic' user after upgrading from 7.17.1 to 8.3.2

[gerib (Gerold 'Geri' Broser)]

Our Cerebro shows the cluster in green, all of our 9 nodes are there with ES: 8.3.2.

There are no [ERROR]s in Kibana's log, just [INFO ]s and [WARN ]s.

This appeared in the process of upgrading our dev cluster and we will not upgrade our prod cluster unless this works properly there. ATM we have 2,732 indices, 5,465 shards, 3,826,484,223 docs, 2.21 TB in our prod cluster, so we want to be absolutely sure that everything works before upgrading it.

[Yang_Wang]

You need grab logs from the Elasticsearch node that handles the Kibana login to see the underlying issue.

Also, are you able to login with any other users? Is the problem just with the elastic user?

[gerib (Gerold 'Geri' Broser)]

I checked the log of the current master node (Is this the one that handles Kibana logins?). The only lines with "error" are:

Line 1: 2022-09-19 10:08:38,122 main INFO Cannot initialize scripting support because this JRE does not support it. java.lang.NoClassDefFoundError: javax/script/ScriptEngineManager
Line 291: 	at org.elasticsearch.indices.recovery.RecoverySourceHandler$OperationBatchSender.handleError(RecoverySourceHandler.java:1209) ~[elasticsearch-8.3.2.jar:?]
Line 292: 	at org.elasticsearch.indices.recovery.RecoverySourceHandler$OperationBatchSender.handleError(RecoverySourceHandler.java:1126) ~[elasticsearch-8.3.2.jar:?]

The msg on line 1 appears also in the logs of our other two master nodes.

users_roles contains:

...
admin:cerebro
kibana_system:kibanaserver
...

Trying:

both lead to:

[elasticforme (Sachin)]

can you connect to cluster via curl

curl -u elastic:password -XGET hostname:9200/_cluster/health?pretty

[gerib (Gerold 'Geri' Broser)]

curl -u elastic:password -XGET hostname:9200/_cluster/health?pretty

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server

[elasticforme (Sachin)]

did you replace password and hostname with your value?

[gerib (Gerold 'Geri' Broser)]

Of course.

[elasticforme (Sachin)]

this tells me you are able to connect because if you are not then you get error like this

Check cluster log and Kibana log closely.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
}

[stephenb (Stephen Brown)]

@gerib

curl: (52) Empty reply from server

This often means that you are trying to connect to an HTTPS endpoint over HTTP

Try this

curl -k -u elastic:password -XGET https://hostname:9200/_cluster/health?pretty

[Justin_Cranford]

W.R.T. the kibanaserver and cerebro errors in Kibana UI:

1. The role kibana_system is only for background administration on an Elasticsearch cluster. That role contains internal-only privileges, no user-level permissions to access the UI. You will need to use a different role for Kibana UI login.

2. The role admin sounds like it might be custom. Check if it has the appropriate privileges as mentioned in Kibana 8.3.2 docs. Privileges may have changed between 7.17.1 and 8.3.2.

[Yang_Wang]

Given the permission error you are having when logging into Kibana, it is likely Elasticsearch authentication/authorization system is working fine. To solve your original problem, you can give user cerebro the superuser role temporarily. Login with it and change the password for the elastic user. You can then remove revert the changes to user cerebro after it is done. To give user cerebro the superuser role, just add the following line to the users_roles file:

superuser:cerebro

PS: Once you made the change, the ES node will automoatically reload the file in at most 5 seconds.