New cluster setup fails

Elastic Stack Elasticsearch
1

I have setup cluster in past without must problem. this time it is just failing don't understand why

Node leaves and joins fine as I test it out. but can't run reset password command for elastic

this is brand new cluster and I have no idea what the password is

/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
output of this command is

Failed to determine the health of the cluster. Cluster health is currently RED.
This means that some cluster data is unavailable and your cluster is not fully functional.
The cluster logs (https://www.elastic.co/guide/en/elasticsearch/reference/8.16/logging.html) might contain information/indications for the underlying cause
It is recommended that you resolve the issues with your cluster before continuing
It is very likely that the command will fail when run against an unhealthy cluster.

If you still want to attempt to execute this command against an unhealthy cluster, you can pass the `-f` parameter.

ERROR: Failed to determine the health of the cluster. Cluster health is currently RED., with exit code 69

Everytime when I try to rest password following message shows up in log

[2025-05-29T13:28:23,940][INFO ][o.e.x.s.a.f.FileUserRolesStore] [houelkdev11] users roles file [/etc/elasticsearch/users_roles] changed. updating users roles...
[2025-05-29T13:28:28,948][INFO ][o.e.x.s.a.f.FileUserPasswdStore] [houelkdev11] users file [/etc/elasticsearch/users] changed. updating users...
[2025-05-29T13:28:28,951][INFO ][o.e.x.s.a.f.FileUserRolesStore] [houelkdev11] users roles file [/etc/elasticsearch/users_roles] changed. updating users roles...

This file is empty and has proper user and permission

when I try with -f following shows up on elastic log
it can't start up .ds-logs-deprecation index.

[2025-05-29T13:34:07,639][INFO ][o.e.c.m.MetadataCreateIndexService] [houelkdev12] [.security-7] creating index, cause [api], templates [], shards [1]/[1]
[2025-05-29T13:34:07,650][INFO ][o.e.c.r.a.AllocationService] [houelkdev12] updating number_of_replicas to [0] for indices [.security-7]
[2025-05-29T13:35:12,225][ERROR][o.e.x.d.l.DeprecationIndexingComponent] [houelkdev12] Bulk write of deprecation logs encountered some failures: [[lklSHZcBTejLPQKNwZBn org.elasticsearch.action.UnavailableShardsException: [.ds-.logs-deprecation.elasticsearch-default-2025.05.29-000001][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.ds-.logs-deprecation.elasticsearch-default-2025.05.29-000001][0]] containing [index {[.logs-deprecation.elasticsearch-default][lklSHZcBTejLPQKNwZBn], source[{"@timestamp":"2025-05-29T18:34:07.165Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.event.category":"indices","event.code":"dot-prefix","message":"Index [.security-7] name begins with a dot (.), which is deprecated, and will not be allowed in a future Elasticsearch version." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[houelkdev12][transport_worker][T#5]","log.logger":"org.elasticsearch.deprecation.validation.DotPrefixValidator","elasticsearch.cluster.uuid":"MISZKmNHS8qOwW22csVrDA","elasticsearch.node.id":"crnKNEoVRieZtnHmCBrt7g","elasticsearch.node.name":"houelkdev12","elasticsearch.cluster.name":"HouElkProd_dev"}
]}]]]]```
2

Whats output of ls -l /etc/elasticsearch/users?

But

If nothing to lose, just start again? How are you installing? When you install it usually has a screenful of information at rpm/deb install time where it tells you the generated elastic password.

3

/etc/elasticsearch/users is blank file

I did restarted many time over by deleting whole data path dir, deleting elasticsearch.keystore file. Deleted everything from /etc/elasticsearch as welll.

this is debian, just use dpkg -i

here is my config. I have been using this certificate to other cluster without issue.

cluster.name: HouElkProd_dev
node.name: houelkdev11
node.roles: [master,ingest,transform]

path.data: /s1/elasticsearch
path.logs: /s1/log/elasticsearch
network.host: 10.21.226.93
http.port: 9200
discovery.seed_hosts: ["houelkdev11","houelkdev12","houekdev13"]
discovery.find_peers_interval: 5s
path.repo: /snapshot
action.destructive_requires_name: true


xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/config/elastic-certificates.p12
4

The reason I asked is I opened as bug when some file in /etc/elasticsearch/... got wrong permission, lost group-writeable bit, which led to issues. I dont have Debian, but on my Ubuntu:

$ sudo ls -l /etc/elasticsearch | grep users
-rw-rw---- 1 root elasticsearch     0 Jan  2 19:55 users
-rw-rw---- 1 root elasticsearch     0 Jan  2 19:55 users_roles

which is correct.

Again, I would anticipate that on a blank system the default elastic user password is set and displayed in the post install. In fact I just checked, the 8.18.1 postinst script has sec ion shown below - something here is not adding up.

node.roles: [master,ingest,transform]

Someone should maybe have a data role ?

$ ls -l elasticsearch_8.18.1_amd64.deb
-rw-r--r-- 1 root root 647923260 May  6 12:22 elasticsearch_8.18.1_amd64.deb
$ sha256sum elasticsearch_8.18.1_amd64.deb
11f78bcf5fab47afedd233ea0abdff1584c0ee7238b012d9ba99bd98f5cae96d  elasticsearch_8.18.1_amd64.deb
$ dpkg-deb -R ~/elasticsearch_8.18.1_amd64.deb tmp
$ less tmp/DEBIAN/postinst
...
if [ "x$IS_UPGRADE" != "xtrue" ]; then
    # Don't exit immediately on error, we want to hopefully print some helpful banners
    set +e
    # Attempt to auto-configure security, this seems to be an installation
    if CLI_NAME="auto-configure-node" \
    CLI_LIBS="modules/x-pack-core,modules/x-pack-security,lib/tools/security-cli" \
    /usr/share/elasticsearch/bin/elasticsearch-cli <<< ""; then
        chown root:elasticsearch "${ES_PATH_CONF}"/certs/http.p12
        chown root:elasticsearch "${ES_PATH_CONF}"/certs/http_ca.crt
        chown root:elasticsearch "${ES_PATH_CONF}"/certs/transport.p12
        if INITIAL_PASSWORD=$(CLI_NAME=auto-config-gen-passwd \
        CLI_LIBS="modules/x-pack-core,modules/x-pack-security,lib/tools/security-cli" \
        /usr/share/elasticsearch/bin/elasticsearch-cli); then
            echo "--------------------------- Security autoconfiguration information ------------------------------"
            echo
            echo "Authentication and authorization are enabled."
            echo "TLS for the transport and HTTP layers is enabled and configured."
            echo
            echo "The generated password for the elastic built-in superuser is : ${INITIAL_PASSWORD}"
            echo
            echo "If this node should join an existing cluster, you can reconfigure this with"
            echo "'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'"
            echo "after creating an enrollment token on your existing cluster."
            echo
            echo "You can complete the following actions at any time:"
            echo
            echo "Reset the password of the elastic built-in superuser with "
            echo "'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'."
            echo
            echo "Generate an enrollment token for Kibana instances with "
            echo " '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'."
            echo
            echo "Generate an enrollment token for Elasticsearch nodes with "
            echo "'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'."
            echo
            echo "-------------------------------------------------------------------------------------------------"
        fi
1 Like
5

yep LOL LOL. how did I missed it. it is working now.

Thank you