Error authenticating user: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST

Hello, I'm climbing trying a cluster without elastic.
After installing elasticsearch and editing the elasticsearch.yml file, I start each machine in the cluster.
However, when doing a curl to check the cluster I receive this error.
The password I am using is correct.

![Erro Elastic|690x374](upload://gnDjIgGKHc8x31fiCiteQFy7VPP.png)
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Basic realm=\"security\", charset=\"UTF-8\"",
            "Bearer realm=\"security\"",
            "ApiKey"
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Basic realm=\"security\", charset=\"UTF-8\"",
        "Bearer realm=\"security\"",
        "ApiKey"
      ]
    }
  },
  "status" : 401
}

My elasticsearch.yml file looks like this:

------------- elasticsearch.yml

cluster.name: elk-cluster
node.name: elk-master-01.environment.int
node.roles: [ master, remote_cluster_client]
network.host: 0.0.0.0
http.port: 9200
discovery.seed_providers: file
cluster.initial_master_nodes: ["elk-master-01.environment.int","elk-master-02.environment.int","elk-master-03.environment.int"]

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/p-elk.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/p-elk.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca/ca.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  /etc/elasticsearch/certs/p-elk.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/p-elk.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca/ca.crt" ]

The cluster log looks like this:

[2025-02-27T02:28:29,309][INFO ][o.e.x.s.a.TokenService   ] [elk-master-01.environment.int] refresh keys
[2025-02-27T02:28:29,598][INFO ][o.e.x.s.a.TokenService   ] [elk-master-01.environment.int] refreshed keys
[2025-02-27T02:28:29,676][INFO ][o.e.x.s.a.Realms         ] [elk-master-01.environment.int] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2025-02-27T02:28:29,681][INFO ][o.e.l.ClusterStateLicenseService] [elk-master-01.environment.int] license [1d71782d-d019-481c-969f-c4ce49bce2f8] mode [basic] - valid
[2025-02-27T02:28:29,699][INFO ][o.e.h.AbstractHttpServerTransport] [dataprod-elk-master-01.environment.int] publish_address {10.47.150.40:9200}, bound_addresses {0.0.0.0:9200}
[2025-02-27T02:28:29,766][INFO ][o.e.n.Node               ] [elk-master-01.environment.int] started {elk-master-01.environment.int}{vq70NQJ6Sei-OFSrZuTDYQ}{E7vXIwkeQdqrhIauLvj78A}{elk-master-01.environment.int}{10.47.150.40}{10.47.150.40:9300}{mr}{8.17.2}{7000099-8521000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}
[2025-02-27T02:28:29,775][INFO ][o.e.n.j.JdkPosixCLibrary ] [elk-master-01.environment.int] Sending 7 bytes to socket
[2025-02-27T02:29:13,644][ERROR][o.e.x.s.a.e.ReservedRealm] [elk-master-01.environment.int] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
[2025-02-27T02:29:13,665][INFO ][o.e.x.s.a.RealmsAuthenticator] [elk-master-01.environment.int] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

This is the cause of your problem.

It might be that your cluster is unhealthy, which is preventing you from logging in.
Or, it might be that you tried too quickly after the node started up (and we haven't loaded your data from disk yet).

Can you wait a few minutes after the node starts, then try, and show us the log file from that failure?

[2025-02-27T03:46:48,026][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elk-master-01.environment.int] [controller/70541] [Main.cc@123] controller (64 bit): Version 8.17.2 (Build 002b4653bd15a4) Copyright (c) 2025 Elasticsearch BV
[2025-02-27T03:46:48,668][INFO ][o.e.x.o.OTelPlugin       ] [elk-master-01.environment.int] OTel ingest plugin is enabled
[2025-02-27T03:46:48,754][INFO ][o.e.x.c.t.YamlTemplateRegistry] [elk-master-01.environment.int] OpenTelemetry index template registry is enabled
[2025-02-27T03:46:48,764][INFO ][o.e.t.a.APM              ] [elk-master-01.environment.int] Sending apm metrics is disabled
[2025-02-27T03:46:48,764][INFO ][o.e.t.a.APM              ] [elk-master-01.environment.int] Sending apm tracing is disabled
[2025-02-27T03:46:48,851][INFO ][o.e.x.s.Security         ] [elk-master-01.environment.int] Security is enabled
[2025-02-27T03:46:50,952][INFO ][o.e.x.s.a.s.FileRolesStore] [-elk-master-01.environment.int] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2025-02-27T03:46:51,980][INFO ][o.e.x.w.Watcher          ] [elk-master-01.environment.int] Watcher initialized components at 2025-02-27T03:46:51.979Z
[2025-02-27T03:46:52,188][INFO ][o.e.x.p.ProfilingPlugin  ] [elk-master-01.environment.int] Profiling is enabled
[2025-02-27T03:46:52,235][INFO ][o.e.x.p.ProfilingPlugin  ] [elk-master-01.environment.int] profiling index templates will not be installed or reinstalled
[2025-02-27T03:46:52,262][INFO ][o.e.x.a.APMPlugin        ] [elk-master-01.environment.int] APM ingest plugin is enabled
[2025-02-27T03:46:52,392][INFO ][o.e.x.c.t.YamlTemplateRegistry] [elk-master-01.environment.int] apm index template registry is enabled
[2025-02-27T03:46:53,828][INFO ][o.e.t.n.NettyAllocator   ] [elk-master-01.environment.int] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2025-02-27T03:46:53,968][INFO ][o.e.d.DiscoveryModule    ] [elk-master-01.environment.int] using discovery type [multi-node] and seed hosts providers [settings, file]
[2025-02-27T03:46:57,513][INFO ][o.e.n.Node               ] [elk-master-01.environment.int] initialized
[2025-02-27T03:46:57,519][INFO ][o.e.n.Node               ] [elk-master-01.environment.int] starting ...
[2025-02-27T03:46:57,537][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [elk-master-01.environment.int] deprecation component started
[2025-02-27T03:46:57,689][INFO ][o.e.t.TransportService   ] [elk-master-01.environment.int] publish_address {10.47.150.10:9300}, bound_addresses {0.0.0.0:9300}
[2025-02-27T03:46:58,446][INFO ][o.e.b.BootstrapChecks    ] [elk-master-01.environment.int] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2025-02-27T03:46:58,581][WARN ][o.e.c.c.ClusterBootstrapService] [elk-master-01.environment.int] this node is locked into cluster UUID [spfmoRlPRbmIFiDpbT5KqQ] but [cluster.initial_master_nodes] is set to [elk-master-01.environment.int, elk-master-02.environment.int, elk-master-03.environment.int]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/important-settings.html#initial_master_nodes
[2025-02-27T03:47:00,995][INFO ][o.e.c.s.ClusterApplierService] [elk-master-01.environment.int] master node changed {previous [], current [{elk-master-02.environment.int}{Yj_Mgxl6ReWtujIQF7dkCg}{Eed4Vh8bSQe6Zw_iXAt3lQ}{elk-master-02.environment.int}{10.47.241.66}{10.47.241.66:9300}{mr}{8.17.2}{7000099-8521000}]}, added {{elk-master-02.environment.int}{Yj_Mgxl6ReWtujIQF7dkCg}{Eed4Vh8bSQe6Zw_iXAt3lQ}{elk-master-02.environment.int}{10.47.241.66}{10.47.241.66:9300}{mr}{8.17.2}{7000099-8521000}}, term: 6, version: 88, reason: ApplyCommitRequest{term=6, version=88, sourceNode={elk-master-02.environment.int}{Yj_Mgxl6ReWtujIQF7dkCg}{Eed4Vh8bSQe6Zw_iXAt3lQ}{elk-master-02.environment.int}{10.47.241.66}{10.47.241.66:9300}{mr}{8.17.2}{7000099-8521000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}}
[2025-02-27T03:47:01,035][INFO ][o.e.h.AbstractHttpServerTransport] [elk-master-01.environment.int] publish_address {10.47.150.10:9200}, bound_addresses {0.0.0.0:9200}
[2025-02-27T03:47:01,088][INFO ][o.e.n.Node               ] [elk-master-01.environment.int] started {elk-master-01.environment.int}{vq70NQJ6Sei-OFSrZuTDYQ}{e4eKyL3TS5yMh0_NvokafA}{elk-master-01.environment.int}{10.47.150.10}{10.47.150.10:9300}{mr}{8.17.2}{7000099-8521000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}
[2025-02-27T03:47:01,098][INFO ][o.e.n.j.JdkPosixCLibrary ] [elk-master-01.environment.int] Sending 7 bytes to socket
[2025-02-27T03:47:02,177][INFO ][o.e.x.s.a.Realms         ] [elk-master-01.environment.int] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2025-02-27T03:47:02,191][INFO ][o.e.l.ClusterStateLicenseService] [elk-master-01.environment.int] license [1d71782d-d019-481c-969f-c4ce49bce2f8] mode [basic] - valid
[2025-02-27T03:47:02,732][INFO ][o.e.x.s.a.TokenService   ] [elk-master-01.environment.int] refresh keys
[2025-02-27T03:47:03,021][INFO ][o.e.x.s.a.TokenService   ] [elk-master-01.environment.int] refreshed keys
[2025-02-27T03:47:20,845][INFO ][o.e.c.s.ClusterApplierService] [elk-master-01.environment.int] added {{elk-master-03.environment.int}{NyPjDSErSSCRTQpW8cVIyQ}{Z3AIZl9aQzyToJNWnKRyXA}{elk-master-03.environment.int}{10.47.1.67}{10.47.241.67:9300}{mr}{8.17.2}{7000099-8521000}}, term: 6, version: 95, reason: ApplyCommitRequest{term=6, version=95, sourceNode={elk-master-02.environment.int}{Yj_Mgxl6ReWtujIQF7dkCg}{Eed4Vh8bSQe6Zw_iXAt3lQ}{elk-master-02.environment.int}{10.47.241.66}{10.47.241.66:9300}{mr}{8.17.2}{7000099-8521000}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}}

I restarted the nodes, and this is my last logs

Your elasticsearch.yml lists 3 nodes as the initial master nodes, is this configuration the same in all 3 nodes?

If so, it is wrong, your node.roles does not have the data role.

node.roles: [ master, remote_cluster_client]

Without a node that can hold data the security auto configuration will not be able to run.

Also, explicitly setting xpack.security.enabled: true in elasticsearch.yml will also make the security auto configuration to not run.

You would need to add data role to your node.roles, restart your nodes and manually reset the elastic user password following this documentation.

Basically running

/path/to/elasticsearch/bin/elasticsearch-reset-password -u elastic
1 Like

Yes, all the 3 nodes have the same configuration in elasticsearch.yml

It worked, thank you very much!!

I trying to add another node, but its returned this logs:

[2025-02-27T05:38:09,026][WARN ][o.e.d.PeerFinder         ] [elk-dashdb-01.environment.int] address [127.0.0.1:9301], node [unknown] discovery result: [][127.0.0.1:9301] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9301: Connection refused: /127.0.0.1:9301: Connection refused; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html
[2025-02-27T05:38:09,026][WARN ][o.e.d.PeerFinder         ] [10.47.150.10elk-dashdb-01.environment.int] address [10.47.241.66:9300], node [unknown] discovery result: [][10.47.241.66:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.47.241.66:9300: Connection refused: /10.47.241.66:9300: Connection refused; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html
[2025-02-27T05:38:09,026][WARN ][o.e.d.PeerFinder         ] [10.47.150.10elk-dashdb-01.environment.int] address [10.47.241.65:9300], node [unknown] discovery result: [][10.47.241.65:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.47.241.65:9300: Connection refused: /10.47.241.65:9300: Connection refused; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html
[2025-02-27T05:38:09,027][WARN ][o.e.d.PeerFinder         ] [10.47.150.10elk-dashdb-01.environment.int] address [10.47.241.67:9300], node [unknown] discovery result: [][10.47.241.67:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.47.241.67:9300: Connection refused: /10.47.241.67:9300: Connection refused; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html
[2025-02-27T05:38:09,036][WARN ][o.e.d.PeerFinder         ] [elk-dashdb-01.environment.int] address [127.0.0.1:9300], node [unknown] discovery result: [10.47.150.10elk-dashdb-01.environment.int][10.47.241.33:9300] successfully discovered local node {10.47.150.10elk-dashdb-01.environment.int}{R_DOSq01Rg63m1_POqo7qA}{yBYsJfqDTKSo69ki-M6vcA}{10.47.150.10elk-dashdb-01.environment.int}{10.47.241.33}{10.47.241.33:9300}{dir}{8.17.2}{7000099-8521000} at [127.0.0.1:9300]; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html

The elasticsearch.yml:

cluster.name: elk-cluster

node.name: elk-dashdb-01.environment.int
node.roles: [ data, ingest, remote_cluster_client]

network.host: 0.0.0.0

http.port: 9200

discovery.seed_providers: file

cluster.initial_master_nodes: ["elk-master-01.environment.int","elk-master-02.environment.int","master-03.environment.int"]

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elk-dashdb.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elk-dashdb.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca/ca.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  /etc/elasticsearch/certs/elk-dashdb.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elk-dashdb.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca/ca.crt" ]