Kibana doesn't start properly at upgrade from 7.17.1 to 8.3.2

[gerib (Gerold 'Geri' Broser)]

Log message(s):

...
... [ERROR][savedobjects-service] [.kibana] Action failed with
  'security_exception: [security_exception] Reason: action [indices:admin/create]
  is unauthorized for user [kibanaserver] with roles [admin] on restricted indices
  [.kibana_8.3.2_001], this action is granted by the index privileges
  [create_index,manage,all]'. ...
...
... [ERROR][savedobjects-service] [.kibana_task_manager] Action failed with
   'security_exception: [security_exception] Reason: action [indices:admin/create]
   is unauthorized for user [kibanaserver] with roles [admin] on restricted indices
   [.kibana_task_manager_8.3.2_001], this action is granted by the index privileges
   [create_index,manage,all]'. ...
...

[Line breaks/spaces added here for readability.]
users_roles contains:

...
admin:kibanaserver
...

roles.yml contains:

admin:
  cluster:
    - all
  indices:
    - names: '*'
      privileges:
        - all
...

I also tried in roles.yml:

...
    - names: [ '*', '.*' ]
...

to no avail.

[gerib (Gerold 'Geri' Broser)]

Solved it. There were different users_roles and roles.yml in the configs of our 9 nodes across 3 servers.

But now the elastic user's password has been reset to ???. And Setting passwords for native and built-in users doesn't work.

I'm feeling like Indiana Jones with this upgrade: one trap door after the other.

[LeeDr (Lee Drengenberg)]

Glad you solved it! Yes, your configs need to match across nodes of the same cluster.

[gerib (Gerold 'Geri' Broser)]

The errors occurred again during one of the next (re-)starts and it persisted until we changed users_roles on all 9 nodes from:

admin:...,kibanaserver,...

to:

admin:...
kibana_system:kibanaserver

It remains a mystery to us why it worked when restarting Kibana a few times and suddenly it didn't persistently – with the admin role (and its settings in roles.yml) assigned to kibanaserver. Just a guess: Perhaps an issue of the order of restarting ES nodes and which master node was the previous current master?