Failed to retrieve password hash for reserved user [kibana]

[niudaye123]

I have to repurpose all the nodes in my cluster, during that process I accidentally deleted the _security index which is for built in user information. Now in the log it said failed to retrieve password hash for reserved user [kibana].

I don't have snapshot made, how do I restore or rebuild this index so I can get kibana up?

thanks!

[ikakavas (Ioannis Kakavas)]

Hi there

@TimV's post from a couple of years back is still relevant: Accidentally deleted .security index for x-pack - #3 by TimV. Note that step 2

From a security point of view, you now have a "new" cluster, so the reserved users elastic and kibana will have reset to their default "changeme" passwords.

doesn't apply anymore since we do not have default passwords for the built-in users, but the rest of the instructions are good to go !

[niudaye123]

Thanks, I saw that post too, but when I tried to re-run the set password command, it gave me the following error. Did I miss any step before running this command?

[root@awselsprdlap01 bin]# ./elasticsearch-setup-passwords interactive

Failed to authenticate user 'elastic' against http://10.172.128.33:9200/_security/_authenticate?pretty
Possible causes include:

• The password for the 'elastic' user has already been changed on this cluster
• Your elasticsearch node is running against a different keystore
  This tool used the keystore at /etc/elasticsearch/esnode01/elasticsearch.keystore

ERROR: Failed to verify bootstrap password

[niudaye123]

Hi, Please see the error I am getting, how do I resolve this? Should I delete the existing bootstrap password from keystore?

[ikakavas (Ioannis Kakavas)]

What's the output of

bin/elasticsearch-keystore list

?

You would need to

1. set the bootstrap.password to something with

   bin/elasticsearch-keystore -f add bootstrap.password
   

2. Restart the node, as the keystore is read on startup and changes to its values are not refreshed on runtime.

3. Run

    bin/elasticsearch-setup-passwords interactive
   

4. You don't need the bootstrap.password anymore so you can remove it with

   bin/elasticsearch-keystore remove bootstrap.password
   

[niudaye123]

1. I set password for bootstrap.password per step 1. you provided. See below:

[root@awselsprdlap01 elasticsearch]# bin/elasticsearch-keystore list
bootstrap.password
keystore.seed
xpack.security.authc.realms.ldap.ldap1.secure_bind_password

2. Restarted nodes.

3. Still failed to start, see below:
   [root@awselsprdlap01 elasticsearch]# bin/elasticsearch-setup-passwords interactive

Failed to authenticate user 'elastic' against http://10.172.128.33:9200/_security/_authenticate?pretty
Possible causes include:

• The password for the 'elastic' user has already been changed on this cluster
• Your elasticsearch node is running against a different keystore
  This tool used the keystore at /etc/elasticsearch/esnode01/elasticsearch.keystore

ERROR: Failed to verify bootstrap password

[niudaye123]

Should I delete the keystore.seed? I believe that's the old password for bootstrap.password.