Missing authentication credentials for REST request

Hello Elastic Users,

I am trying to check if my Elasticsearch is working fine and I got the following problem:

and Im trying to change elastic super-user password, I got this error :

root@Master-node:/usr/share/elasticsearch/bin# ./elasticsearch-setup-passwords auto

Unexpected response code [405] from calling GET http://10.0.2.198:9200/_security/_authenticate?pretty
It doesn't look like the X-Pack security feature is enabled on this Elasticsearch node.
Please check if you have enabled X-Pack security in your elasticsearch.yml configuration file.


ERROR: X-Pack Security is disabled by configuration.
root@Master-node:/usr/share/elasticsearch/bin# nano /etc/elasticsearch/elasticsearch.yml
root@Master-node:/usr/share/elasticsearch/bin# systemctl restart elasticsearch.service
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
root@Master-node:/usr/share/elasticsearch/bin# nano /etc/elasticsearch/elasticsearch.yml
root@Master-node:/usr/share/elasticsearch/bin# systemctl restart elasticsearch.service
root@Master-node:/usr/share/elasticsearch/bin# ./elasticsearch-setup-passwords auto

Failed to authenticate user 'elastic' against http://10.0.2.198:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /etc/elasticsearch/elasticsearch.keystore

You can use the `elasticsearch-reset-password` CLI tool to reset the password of the 'elastic' user


ERROR: Failed to verify bootstrap password
root@Master-node:/usr/share/elasticsearch/bin# ./elasticsearch-reset-password -u elastic

ERROR: Failed to determine the health of the cluster. Unexpected http status [503]
root@Master-node:/usr/share/elasticsearch/bin#

my Elasticsearch.yml is the following:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: es-cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: "master-node"
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#node.master : true
#node.data : true
#node.roles : ["master"]
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 10.0.2.198
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.0.2.198", "10.0.2.199", "10.0.2.197"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["master-node"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 02-08-2022 22:52:49
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["Master-node"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Any suggestions for resolve this problem?

Thank you in advance

Can you run elasticsearch-reset-password in verbose mode, i.e. with -v. Also it would be helpful if you could share the server logs when running the CLI.

Also please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

Hello Mr Yang,
Thank you so much for your reply.

When I run elasticsearch-reset-password in verbose mode, I get the following :

root@Master-node:/usr/share/elasticsearch/bin# ./elasticsearch-reset-password -u elastic -v
Unexpected http status [401] while attempting to determine cluster health. Will retry at most 5 more times.
Unexpected http status [401] while attempting to determine cluster health. Will retry at most 4 more times.

ERROR: Failed to determine the health of the cluster. Unexpected http status [503]
root@Master-node:/usr/share/elasticsearch/bin# ./elasticsearch-reset-password -u elastic -v --url https://10.0.2.198:9200
Unexpected http status [401] while attempting to determine cluster health. Will retry at most 5 more times.

ERROR: Failed to determine the health of the cluster. Unexpected http status [503]
root@Master-node:/usr/share/elasticsearch/bin#

and these logs are from /var/log/elasticsearch/es-cluster.log :

[2022-08-04T08:39:25,750][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [master-node] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.0.2.198:9300, remoteAddress=/10.0.2.199:41762, profile=default}
[2022-08-04T08:39:26,484][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [master-node] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.2.198:9300, remoteAddress=/10.0.2.197:34616, profile=default}
[2022-08-04T08:39:26,506][WARN ][o.e.d.PeerFinder         ] [master-node] address [10.0.2.197:9300], node [null], requesting [false] discovery result: [][10.0.2.197:9300] general node connection failure: handshake failed because connection reset
[2022-08-04T08:39:26,520][WARN ][o.e.c.s.DiagnosticTrustManager] [master-node] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=hot-node], fingerprint [521c845b2b69e3d7077c682be72a58c8cfa96a88], no keyUsage and no extendedKeyUsage; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [d33bb6815ac2612591b3cb1a2cca3e42362fb24c]) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl (with trust configuration: StoreTrustConfig{path=certs/transport.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elasticsearch security auto-configuration HTTP CA] but the trusted certificate has fingerprint [4711584bbe3fe99b9217436fe9e9fbb9d4724c39]
[2022-08-04T08:39:26,521][WARN ][o.e.t.OutboundHandler    ] [master-node] send message failed [channel: Netty4TcpChannel{localAddress=/10.0.2.198:33170, remoteAddress=/10.0.2.199:9300, profile=default}]
[2022-08-04T08:39:26,751][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [master-node] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.0.2.198:9300, remoteAddress=/10.0.2.199:41768, profile=default}
[2022-08-04T08:39:27,484][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [master-node] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.2.198:9300, remoteAddress=/10.0.2.197:34620, profile=default}
[2022-08-04T08:39:27,506][WARN ][o.e.d.PeerFinder         ] [master-node] address [10.0.2.197:9300], node [null], requesting [false] discovery result: [][10.0.2.197:9300] general node connection failure: handshake failed because connection reset
[2022-08-04T08:39:27,520][WARN ][o.e.c.s.DiagnosticTrustManager] [master-node] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=hot-node], fingerprint [521c845b2b69e3d7077c682be72a58c8cfa96a88], no keyUsage and no extendedKeyUsage; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [d33bb6815ac2612591b3cb1a2cca3e42362fb24c]) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl (with trust configuration: StoreTrustConfig{path=certs/transport.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elasticsearch security auto-configuration HTTP CA] but the trusted certificate has fingerprint [4711584bbe3fe99b9217436fe9e9fbb9d4724c39]
[2022-08-04T08:39:27,521][WARN ][o.e.t.OutboundHandler    ] [master-node] send message failed [channel: Netty4TcpChannel{localAddress=/10.0.2.198:33174, remoteAddress=/10.0.2.199:9300, profile=default}]
[2022-08-04T08:39:27,522][WARN ][o.e.d.PeerFinder         ] [master-node] address [10.0.2.199:9300], node [null], requesting [false] discovery result: [][10.0.2.199:9300] general node connection failure: handshake failed because connection reset
[2022-08-04T08:39:27,522][WARN ][o.e.t.TcpTransport       ] [master-node] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.0.2.198:33174, remoteAddress=/10.0.2.199:9300, profile=default}], closing connection
[2022-08-04T08:39:27,751][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [master-node] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.0.2.198:9300, remoteAddress=/10.0.2.199:41770, profile=default}
[2022-08-04T08:39:28,484][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [master-node] received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.0.2.198:9300, remoteAddress=/10.0.2.197:34624, profile=default}
[2022-08-04T08:39:28,506][WARN ][o.e.d.PeerFinder         ] [master-node] address [10.0.2.197:9300], node [null], requesting [false] discovery result: [][10.0.2.197:9300] general node connection failure: handshake failed because connection reset
[2022-08-04T08:39:28,519][WARN ][o.e.c.s.DiagnosticTrustManager] [master-node] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=hot-node], fingerprint [521c845b2b69e3d7077c682be72a58c8cfa96a88], no keyUsage and no extendedKeyUsage; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [d33bb6815ac2612591b3cb1a2cca3e42362fb24c]) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl (with trust configuration: StoreTrustConfig{path=certs/transport.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elasticsearch security auto-configuration HTTP CA] but the trusted certificate has fingerprint [4711584bbe3fe99b9217436fe9e9fbb9d4724c39

Thank you again !

Your cluster did not form up correctly because of SSL certificate issues. This logging line is the key

[2022-08-04T08:39:28,519][WARN ][o.e.c.s.DiagnosticTrustManager] [master-node] failed to establish trust with server at []; the server provided a certificate with subject name [CN=hot-node], fingerprint [521c845b2b69e3d7077c682be72a58c8cfa96a88], no keyUsage and no extendedKeyUsage; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [d33bb6815ac2612591b3cb1a2cca3e42362fb24c]) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl (with trust configuration: StoreTrustConfig{path=certs/transport.p12, password=, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elasticsearch security auto-configuration HTTP CA] but the trusted certificate has fingerprint [4711584bbe3fe99b9217436fe9e9fbb9d4724c39

It means the node does not trust the other node's certificate because its issuer is not trusted. How did you configure security for the two nodes? I'd guess that you ran security auto-configuration separately for each of them. This would make each node to start with their separate trust anchors which have the same name but different fingerprints.

After starting up your first node with security auto-configured, to add additional nodes, you should follow this guide
You may need delete the existing certs and keys (if they have been auto-generated previously) on the additional nodes. If you installed them with RPM or DEB package, you can use the elasticsearch-reconfigure-node tool to re-configure a node to join an existing cluster.