Kibana 6.4.2 plugin:security Privileges are missing and can't be removed, currently

[samwato (Sam)]

Was running on Elastic Cloud 6.4.2

Received error when logged into Kibana. No known cause.
plugin:security@6.4.3 | Privileges are missing and can't be removed, currently.

Tried Updating to 6.5.3 to fix error but Kibana instance never starts..
"Waiting until instances are running"

Terminated Kibana a few times and created a new instance. Still no luck.

Removed /.kibana_1 and /.kibana_2 for testing, still same errors.

Unfortunately I cannot try the 6.5.0 known issue fix as using Elastic Cloud.

Currently,
Elasticsearch is sitting on version 6.5.3
Kibana cannot update from 6.4.2 to latest

Any suggestions.

[mark.dueck]

FYI, we wound up solving this:

One of our crazy developers pointed an external Kibana 6.5.3 install at our elasticsearch/kibana 6.4.2 cloud deployment. It seems that the new kibana eagerly started to upgrade .security and .kibana indexes before it reported that elastic was running the wrong version, creating the error in kibana 6.4.2.

We found the code responsible for creating the error (so thanks for making this product open source) and realised we had two options, make the existingPrivileges match expectedPrivileges or just remove the existingPrivileges all together.

The easiest solution was to delete these entries from the .security-6 index which causes kibana to re-create them:
"name":"all"
name":"read"
"name":"space_all"
"name":"space_read"

The top two were in 6.4.2 so first we just tried removing those two (the others are new to 6.5) but whatever is querying these things must pull them all up so we had to get rid of them all. Their _id's all start with "application-privilege_kibana-.kibana:

After that Kibana 6.4.2 was able to start again (and presumably re-created the new privs) which then allowed us to upgrade it (Note: this is all on elastic cloud, we don't have any terminal access).

Hope this helps someone.

[Larry_Gregory]

@mark.dueck thanks for researching and posting your solution! You're absolutely right, having multiple versions of Kibana running against the same .kibana index may cause the older versions to report this error.

For others running into this (specifically, moving from 6.4.x => 6.5.x), running the following should help. Restart Kibana after running these commands, and you should be all set. Note if you don't use the default .kibana index, you'll need to replace .kibana below with the name of your actual kibana index:

DELETE _xpack/security/privilege/kibana-.kibana/space_all
DELETE _xpack/security/privilege/kibana-.kibana/space_read