Failed to retrieve password hash for reserved user [kibana]

I have to repurpose all the nodes in my cluster, during that process I accidentally deleted the _security index which is for built in user information. Now in the log it said failed to retrieve password hash for reserved user [kibana].

I don't have snapshot made, how do I restore or rebuild this index so I can get kibana up?

thanks!

Hi there

@TimV's post from a couple of years back is still relevant: Accidentally deleted .security index for x-pack. Note that step 2

From a security point of view, you now have a "new" cluster, so the reserved users elastic and kibana will have reset to their default "changeme" passwords.

doesn't apply anymore since we do not have default passwords for the built-in users, but the rest of the instructions are good to go !

Thanks, I saw that post too, but when I tried to re-run the set password command, it gave me the following error. Did I miss any step before running this command?

[root@awselsprdlap01 bin]# ./elasticsearch-setup-passwords interactive

Failed to authenticate user 'elastic' against http://10.172.128.33:9200/_security/_authenticate?pretty
Possible causes include:

  • The password for the 'elastic' user has already been changed on this cluster
  • Your elasticsearch node is running against a different keystore
    This tool used the keystore at /etc/elasticsearch/esnode01/elasticsearch.keystore

ERROR: Failed to verify bootstrap password

Hi, Please see the error I am getting, how do I resolve this? Should I delete the existing bootstrap password from keystore?

What's the output of

bin/elasticsearch-keystore list

?

You would need to

  1. set the bootstrap.password to something with

    bin/elasticsearch-keystore -f add bootstrap.password
    
  2. Restart the node, as the keystore is read on startup and changes to its values are not refreshed on runtime.

  3. Run

     bin/elasticsearch-setup-passwords interactive
    
  4. You don't need the bootstrap.password anymore so you can remove it with

    bin/elasticsearch-keystore remove bootstrap.password
    
  1. I set password for bootstrap.password per step 1. you provided. See below:

[root@awselsprdlap01 elasticsearch]# bin/elasticsearch-keystore list
bootstrap.password
keystore.seed
xpack.security.authc.realms.ldap.ldap1.secure_bind_password

  1. Restarted nodes.

  2. Still failed to start, see below:
    [root@awselsprdlap01 elasticsearch]# bin/elasticsearch-setup-passwords interactive

Failed to authenticate user 'elastic' against http://10.172.128.33:9200/_security/_authenticate?pretty
Possible causes include:

  • The password for the 'elastic' user has already been changed on this cluster
  • Your elasticsearch node is running against a different keystore
    This tool used the keystore at /etc/elasticsearch/esnode01/elasticsearch.keystore

ERROR: Failed to verify bootstrap password

Should I delete the keystore.seed? I believe that's the old password for bootstrap.password.