Failed to authenticate user [kibana_system]

When I tried Kibana development, I ran the following commands:

yarn es snapshot --license trial
yarn start --run-examples

Then I encountered an error in Elasticsearch:

failed to retrieve password hash for reserved user [kibana_system]
Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

And an error in Kibana:

[error][savedobjects-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

I tried the default password for user [kibana_system] in kibana.yml and also tried to change the password to "changeme" or "kibanapassword", but they all got errors. Can anyone help me?

It sounds like that you are using the wrong password. Did you set the password yourself or did you get auto-generated passwords from setup-passwords ?

You can test the passwords you think are correct against elasticsearch manually by running

curl -u <user_here> 'http://localhost:9200/_xpack/security/_authenticate?pretty'

replacing <user_here> with the relevant username ( elastic , kibana , logstash_system ) and entering the appropriate password for each.

hope this helps ? Thanks
Rashmi

When I was running Elasticsearch by command yarn es snapshot --license trial, I tried to change all users' passwords to "changeme" by running bin/elasticsearch-setup-passwords interactive in .es/8.0.0 foler in another terminal. However, I encountered the following error:

Connection failure to: http://127.0.0.1:9200/_security/user/apm_system/_password?pretty failed: Read timed out
ERROR: Failed to set password for user [apm_system].

I also tried to use bin/elasticsearch to run Elasticsearch, but I couldn't even change the passwords in this case. The following is the error message:

Failed to authenticate user 'elastic' against http://127.0.0.1:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /Users/sheen/Desktop/kibana/.es/8.0.0/config/elasticsearch.keystore
ERROR: Failed to verify bootstrap password

I tried another branch (7.13) instead of the master branch in Kibana git repo and didn't encounter any errors. I believe the above errors might be caused by bugs in Elasticsearch 8.0.0 snapshot.