Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

Elastic Stack Kibana
1

Hello
When I start elastic, I am getting the error below:
Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
I didn't have this error previously.
When I try to authenticate with curl command, the user can authenticate.
What do I need to update in the config files ?

curl -u elastic:elastic http://localhost:9200
{
  "name" : "DESKTOP-QCJ7Q5R",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "aBpl0CKSTcq4XOzb1aERdA",
  "version" : {
    "number" : "8.7.0",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "09520b59b6bc1057340b55750186466ea715e30e",
    "build_date" : "2023-03-27T16:31:09.816451435Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
2

Hi @aaronlbk

Exactly where / how are you getting that error?

Are you using Kibana can you share your can you share both your elasticsearch.yml and kibana.yml

3

I get the error at the start of elasticsearch.
Everything is commented in elasticsearch.yaml
the configuration kibana.yaml

#
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601



# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://localhost:9200"]


elasticsearch.username: "kibana_system"
elasticsearch.password: "xxxxx"

As I mentionned the error that I got at the start of elastic is:

Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

4

Hi @aaronlbk

Thank you for the information.

We're going to need more to help. Just a simple error message or part of an error message isn't enough for us to under understand exactly what's going on and how to solve it with you.

We ask questions and need answers from you as completely as possible in order for us to help

What changed? Did you upgrade? Did you add a new client (beats, lang client, HTTP client etc)

Apologies I do not know what that means...

Yes, you mentioned that... but that alone is not enough to help solve the issue... you did not even include the entire message.

I asked about kibana.yml as there can be an issue with the credentials there.

If you search google or this forum for that message you will see many results for this error message.

This message occurs when a client is trying to access elasticsearch with improper credentials.... so something is trying to access elasticsearch with a wrong password for the elastic user

I think you are saying that elasticsearch does in fact start and you tested that with curl and it worked.

So you need to look for what else is trying to connect

Kibana, beats, java client something is most likely trying to connect with user elastic and the wrong password.

I just repeated this error by curl to my own cluster with the wrong password for elastic

2023-06-19 14:05:26 {"@timestamp":"2023-06-19T21:05:26.762Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": ...

You can try that with your curl and you will see... so you need to figure out what is trying to connect to your elasticsearch cluster....

5

thank you for your reply

here is the content of elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

the logs after the start of elastic is below

[2023-06-21T07:58:29,053][INFO ][o.e.p.PluginsService     ] [XXXXX] loaded module [x-pack-watcher]
[2023-06-21T07:58:29,054][INFO ][o.e.p.PluginsService     ] [XXXXX] loaded module [x-pack-write-load-forecaster]
[2023-06-21T07:58:29,054][INFO ][o.e.p.PluginsService     ] [XXXXX] no plugins loaded
[2023-06-21T07:58:33,603][WARN ][stderr                   ] [XXXXX] juin 21, 2023 7:58:33 AM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
[2023-06-21T07:58:33,615][INFO ][o.e.e.NodeEnvironment    ] [XXXXX] using [1] data paths, mounts [[Disk 2 (D:)]], net usable_space [158.1gb], net total_space [168.2gb], types [NTFS]
[2023-06-21T07:58:33,616][INFO ][o.e.e.NodeEnvironment    ] [XXXXX] heap size [7.4gb], compressed ordinary object pointers [true]
[2023-06-21T07:58:34,035][INFO ][o.e.n.Node               ] [XXXXX] node name [XXXXX], node ID [0HoJnSfgS5iFGJjeViHp8A], cluster name [elasticsearch], roles [data_hot, ml, data_frozen, ingest, data_cold, data, remote_cluster_client, master, data_warm, data_content, transform]
[2023-06-21T07:58:37,162][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [XXXXX] [controller/13988] [Main.cc@123] controller (64 bit): Version 8.7.0 (Build e4e1c23721e58c) Copyright (c) 2023 Elasticsearch BV
[2023-06-21T07:58:37,681][INFO ][o.e.x.s.Security         ] [XXXXX] Security is enabled
[2023-06-21T07:58:38,177][INFO ][o.e.x.s.a.s.FileRolesStore] [XXXXX] parsed [0] roles from file [D:\programs\ELK\elasticsearch\config\roles.yml]
[2023-06-21T07:58:39,242][INFO ][o.e.x.p.ProfilingPlugin  ] [XXXXX] Profiling is enabled
[2023-06-21T07:58:40,282][INFO ][o.e.t.n.NettyAllocator   ] [XXXXX] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-06-21T07:58:40,317][INFO ][o.e.i.r.RecoverySettings ] [XXXXX] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-06-21T07:58:40,378][INFO ][o.e.d.DiscoveryModule    ] [XXXXX] using discovery type [multi-node] and seed hosts providers [settings]
[2023-06-21T07:58:42,098][INFO ][o.e.n.Node               ] [XXXXX] initialized
[2023-06-21T07:58:42,099][INFO ][o.e.n.Node               ] [XXXXX] starting ...
[2023-06-21T07:58:42,132][INFO ][o.e.x.s.c.f.PersistentCache] [XXXXX] persistent cache index loaded
[2023-06-21T07:58:42,134][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [XXXXX] deprecation component started
[2023-06-21T07:58:42,435][INFO ][o.e.t.TransportService   ] [XXXXX] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2023-06-21T07:58:43,279][WARN ][o.e.b.BootstrapChecks    ] [XXXXX] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2023-06-21T07:58:43,280][WARN ][o.e.b.BootstrapChecks    ] [XXXXX] Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
[2023-06-21T07:58:43,282][INFO ][o.e.c.c.ClusterBootstrapService] [XXXXX] this node is locked into cluster UUID [aBpl0CKSTcq4XOzb1aERdA] and will not attempt further cluster bootstrapping
[2023-06-21T07:58:43,297][INFO ][o.e.c.c.ClusterBootstrapService] [XXXXX] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2023-06-21T07:58:43,462][INFO ][o.e.c.s.MasterService    ] [XXXXX] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {XXXXX}{0HoJnSfgS5iFGJjeViHp8A}{URIGfi6FRzeNaDkk0t2XMw}{XXXXX}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.0} completing election], term: 29, version: 763, delta: master node changed {previous [], current [{XXXXX}{0HoJnSfgS5iFGJjeViHp8A}{URIGfi6FRzeNaDkk0t2XMw}{XXXXX}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.0}]}
[2023-06-21T07:58:43,615][INFO ][o.e.c.s.ClusterApplierService] [XXXXX] master node changed {previous [], current [{XXXXX}{0HoJnSfgS5iFGJjeViHp8A}{URIGfi6FRzeNaDkk0t2XMw}{XXXXX}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.0}]}, term: 29, version: 763, reason: Publication{term=29, version=763}
[2023-06-21T07:58:43,661][INFO ][o.e.r.s.FileSettingsService] [XXXXX] starting file settings watcher ...
[2023-06-21T07:58:43,670][INFO ][o.e.r.s.FileSettingsService] [XXXXX] file settings service up and running [tid=83]
[2023-06-21T07:58:43,678][INFO ][o.e.c.c.NodeJoinExecutor ] [XXXXX] node-join: [{XXXXX}{0HoJnSfgS5iFGJjeViHp8A}{URIGfi6FRzeNaDkk0t2XMw}{XXXXX}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.0}] with reason [completing election]
[2023-06-21T07:58:43,735][INFO ][o.e.h.AbstractHttpServerTransport] [XXXXX] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2023-06-21T07:58:43,736][INFO ][o.e.n.Node               ] [XXXXX] started {XXXXX}{0HoJnSfgS5iFGJjeViHp8A}{URIGfi6FRzeNaDkk0t2XMw}{XXXXX}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.7.0}{ml.allocated_processors_double=8.0, ml.max_jvm_size=8019509248, ml.allocated_processors=8, ml.machine_memory=16037543936, xpack.installed=true}
[2023-06-21T07:58:44,082][INFO ][o.e.x.s.a.RealmsAuthenticator] [XXXXX] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-06-21T07:58:44,218][INFO ][o.e.l.LicenseService     ] [XXXXX] license [a97f561d-90f4-4340-9368-516748521f0f] mode [basic] - valid
[2023-06-21T07:58:44,220][INFO ][o.e.x.s.a.Realms         ] [XXXXX] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-06-21T07:58:44,227][INFO ][o.e.g.GatewayService     ] [XXXXX] recovered [20] indices into cluster_state
[2023-06-21T07:58:44,690][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [XXXXX] Node [{XXXXX}{0HoJnSfgS5iFGJjeViHp8A}] is selected as the current health node.
[2023-06-21T07:58:47,601][INFO ][o.e.c.r.a.AllocationService] [XXXXX] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.apm-source-map][0]]])." previous.health="RED" reason="shards started [[.apm-source-map][0]]"
[2023-06-21T07:59:21,695][INFO ][o.e.x.s.a.RealmsAuthenticator] [XXXXX] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-06-21T08:00:08,011][INFO ][o.e.x.s.a.RealmsAuthenticator] [XXXXX] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-06-21T08:01:05,654][INFO ][o.e.x.s.a.RealmsAuthenticator] [XXXXX] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-06-21T08:02:01,507][INFO ][o.e.x.s.a.RealmsAuthenticator] [XXXXX] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-06-21T08:02:42,501][INFO ][o.e.x.s.a.RealmsAuthenticator] [XXXXX] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

I cannot figure out from this logs if something is trying to connect to elastic

6

Hi @aaronlbk

Something does not make sense you have an empty elasticsearch.yml yet you have authentication enabled ... that is not normal / highly unusual (basically not sure how that is even possible)

Are you sure that is the elasticsearch.yml that is being used?

How did you install elasticsearch? is this an upgrade from 7.x?

Normally you would turn on Audit logging but that requires a License.

Something is definitely attempting to authenticate...

You might be able to see which process by using something like the following since it looks like elasticsearch is only bound to localhost/127.0.0.1

lsof -nPi tcp:9200

Example on my localhost in this case I can see filebeat is connecting, you might need to run it more than once.

$ lsof -nPi tcp:9200
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
filebeat   3424 sbrown   14u  IPv4 0x62479b09038ccbd9      0t0  TCP 127.0.0.1:64616->127.0.0.1:9200 (ESTABLISHED)
com.docke 28413 sbrown  169u  IPv6 0x62479b0904e04771      0t0  TCP *:9200 (LISTEN)
com.docke 28413 sbrown  231u  IPv6 0x62479b0901835771      0t0  TCP 127.0.0.1:9200->127.0.0.1:64616 (ESTABLISHED)
7

I have installed it manually on windows. The version is 8.7.0

8

Not sure what that means... did you install it as a service?

Did you use the auto configuration of security? Or did you disable that?
Did you disable SSL?
The elasticsearch.yml should not be empty

But all that does not really matter... as I explained you have another process trying to log into elasticsearch as the elastic user that is providing the wrong credentials

You are going to need to look at the network connections

9

@aaronlbk

I was looking at something else and saw this

Perhaps something else to look at

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.