Auto-generated deafult password hash error

Elastic Stack Elasticsearch
1

so as i am following Tutorial 1: Installing a self-managed Elastic Stack at the end of step 3 i should be able to run curl command and see some sort of default message that indicate my set up was fine but instead i get the following error:

{"error":{"root_cause":[{"type":"status_exception","reason":"Cluster state has not been recovered yet, cannot write to the [null] index"}],"type":"authentication_processing_error","reason":"failed to promote the auto-configured elastic password hash","caused_by":{"type":"status_exception","reason":"Cluster state has not been recovered yet, cannot write to the [null] index"}},"status":503}

on thing is that i noticed when i try the same command but with a wrong password the error changes to:

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}

and if i set xpack.security.enabled to false it works. I tried resetting password as follow :

/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic

but then i get :

ERROR: Failed to determine the health of the cluster. Unexpected http status [503], with exit code 65

here is my elasticsearch.yml:

cluster.name: elasticsearch-demo
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["localhost"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

and also the log file content:

[2024-08-22T20:55:22,516][WARN ][o.e.x.s.a.RealmsAuthenticator] [node-1] An error occurred while attempting to authenticate [elastic] again> org.elasticsearch.ElasticsearchAuthenticationProcessingError: failed to promote the auto-configured elastic password hash at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationProcessError(Exceptions.java:44) ~[?:?] at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$2(ReservedRealm.java:160) ~[?:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearc> at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:179) ~[elasticsearch-8.14.3.jar:?] . . . Caused by: org.elasticsearch.ElasticsearchStatusException: Cluster state has not been recovered yet, cannot write to the [null] index at org.elasticsearch.xpack.security.support.SecurityIndexManager.prepareIndexIfNeededThenExecute(SecurityIndexManager.java:430) ~[?> ... 67 more [2024-08-22T20:55:22,532][WARN ][r.suppressed ] [node-1] path: /, params: {}, status: 503 org.elasticsearch.ElasticsearchAuthenticationProcessingError: failed to promote the auto-configured elastic password hash at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationProcessError(Exceptions.java:44) ~[?:?] at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$2(ReservedRealm.java:160) ~[?:?] at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearc> at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:179) ~[elasticsearch-8.14.3.jar:?] . . . at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:1570) ~[?:?] Caused by: org.elasticsearch.ElasticsearchStatusException: Cluster state has not been recovered yet, cannot write to the [null] index at org.elasticsearch.xpack.security.support.SecurityIndexManager.prepareIndexIfNeededThenExecute(SecurityIndexManager.java:430) ~[?> ... 67 more

Thanks.

2

Everything you've posted are symptoms, not causes.

This means that your cluster is not in a state where it can read/write data. I cannot tell you what caused that, but it almost certainly has nothing to do with your password setup - the root of the problem is that security cannot do its job if it cannot read and write data from the cluster.

You need to go back further in the log and look for messages that explain why the cluster was unable to recover its state from disk

3

Well i really appreciate your answer, i had this bad habit of only looking for warning and errors in a log file. so i did as you said. there is this line that might tell something:

[INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [node-1] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the the bin/elasticsearch-reset-password tool to set the password for the elastic user

any idea what could cause terminal error? cause anyway at the end of installation it display an auto generated password. and it kinda seems to be to aware of that password cause when i use the right password i get a different error. i really like to know what did i do wrong so i do not do it again but at this point even if i try to reset pass i get health error as stated above in the question.

by the way im running my node on a VM in VMWare workstation, if it has anything to do with it anyway.

4

That's not the problem.

You're looking for something (that probably has nothing to do with security) that explains why your cluster cannot recover data from disk.

You need to work out why your cluster is responding with 503 status (Service Unavailable).
That's a sign that your cluster is not working, and no amount of fiddling with passwords will make it work because it's not a password problem.

7

So I just gonna let this command to be here maybe one day somebody find it useful.

as TImV mentioned above the problem was not related to password at all. i just had to set cluster.initial_master_nodes correctly. check the link for more detail about this parameter in elasticsearch.yml file.