Kibana doesn't start properly at upgrade from 7.17.1 to 8.3.2

Log message(s):

...
... [ERROR][savedobjects-service] [.kibana] Action failed with
  'security_exception: [security_exception] Reason: action [indices:admin/create]
  is unauthorized for user [kibanaserver] with roles [admin] on restricted indices
  [.kibana_8.3.2_001], this action is granted by the index privileges
  [create_index,manage,all]'. ...
...
... [ERROR][savedobjects-service] [.kibana_task_manager] Action failed with
   'security_exception: [security_exception] Reason: action [indices:admin/create]
   is unauthorized for user [kibanaserver] with roles [admin] on restricted indices
   [.kibana_task_manager_8.3.2_001], this action is granted by the index privileges
   [create_index,manage,all]'. ...
...

[Line breaks/spaces added here for readability.]
users_roles contains:

...
admin:kibanaserver
...

roles.yml contains:

admin:
  cluster:
    - all
  indices:
    - names: '*'
      privileges:
        - all
...

I also tried in roles.yml:

...
    - names: [ '*', '.*' ]
...

to no avail.

Solved it. There were different users_roles and roles.yml in the configs of our 9 nodes across 3 servers.

But now the elastic user's password has been reset to ???. And Setting passwords for native and built-in users doesn't work.

I'm feeling like Indiana Jones with this upgrade: one trap door after the other.

Glad you solved it! Yes, your configs need to match across nodes of the same cluster.

The errors occurred again during one of the next (re-)starts and it persisted until we changed users_roles on all 9 nodes from:

admin:...,kibanaserver,...

to:

admin:...
kibana_system:kibanaserver

It remains a mystery to us why it worked when restarting Kibana a few times and suddenly it didn't persistently – with the admin role (and its settings in roles.yml) assigned to kibanaserver. Just a guess: Perhaps an issue of the order of restarting ES nodes and which master node was the previous current master?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.