ES 8.6.2 - puzzling "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]"?

Elastic Stack Elasticsearch
1

I am aware this error has come up before, but please note the version, 8.6.2. Most of the others are about v7. So far I have found the whole configuration of 8.6.2. much more of a challenge (from the security PoV) than v7.

I am currently running 7.10.2 on my machine (w10) as a service (http://localhost:9200), but I have configured things to run 8.6.2 on https://localhost:9500. The idea being to upgrade to 8.6.2. as my working version when problem-free. So up to now, to start the 8.6.2 server, I have just gone, in a terminal (command prompt window):

...\bin>set ES_PATH_CONF=D:/apps/Elasticsearch/elasticsearch-8.6.2/config
...\bin>elasticsearch.bat

... and, after copious output, I get a message about a shard turning from RED to GREEN. And I have been able to access the server, first using a browser, but then from a Python script. The latter also references the "certificate" when using the "requests" module. So I've jumped through all those hoops.

Having generated a password for 8.6.2, and after the browser saved this URL/password combination, for about a week I was able just to start the server as above, and everything worked fine.

What's puzzling now is that, after not having run the 8.6.2 server as above for a few days, I try again, I get the message about the shard going GREEN ... but then my access attempt fails in the browser, with a 401 response, and the above message, "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", in the server terminal (command prompt) window.

I tried rebooting... same failure. I also manually copied over what should be the correct password in the browser (because this is concealed using dots) to try and connect... For what it's worth, I also tried with another browser: no joy.

I had a look at the logs. The only logs under \logs which seems to be being modified is "gc.log", but I've no idea what I might be looking for. Any attempt to try and access the server generates this log message:

[2023-06-04T08:33:44.515+0000][14704][safepoint ] Safepoint "Cleanup", Time since last: 11040257300 ns, Reaching safepoint: 149600 ns, At safepoint: 3000 ns, Total: 152600 ns

... I could at this point try generating a new password: I've taken notes about how to do that. But I don't want to, because I haven't knowingly done anything which might have caused this mess-up and I want ideally to understand and resolve the root problem.

What should I do here? Is there something I should be looking for in the logs? I have the gc.log older files going back about 3 months now.

For that matter, can someone say where the password for a given user (the username I'm using is the default, "elastic", currently) is located in the 8.6.2 setup (on W10)? Maybe I can check that it hasn't become corrupted or something...

I have a vague suspicion: because I have used an application on my machine over the past days which uses my current working ES (i.e. the W10 service, using 7.10.2). Despite having done what I could to make the 8.6.2 configuration as separate as possible from the installed 7.10.2 setup, is it possible that the latter might be messing up the username-passwords of 8.6.2?

2

I decided to reset the password for user "elastic". Everything working.

Then I ran my app which uses the ES 7.10.2 installation and did various things involving the indices there.

Then I started up the 8.6.2 server ... and everything still ran OK on https://localhost:9500.

So it appears that this (i.e. having 2 ESs on the same machine) was not (necessarily) the main cause here. NB things are configured so that they have completely separate storage locations, different cluster.names, node.names, etc.

Still like to know where/how ES stores its user-password pairs.

3

In an index that's usually aliased to .security.

You're best off not using the super user if you can.

5

Thanks... the problem came back. I then may have found a solution based on your clues. I propose leaving this here for anyone experiencing similar problems.

Yesterday I reset the "elastic" account password. And for a time I was able to run and also use the 8.6.2 server. So something I've done today (or something some process has done) has messed things up again.

But I tried just now and it's obviously "lost" the password (i.e. it won't let me log on using it, for example in a browser).

I found this page and created a user. I ran the server... and to no surprise the new account + pwd did not enable me to log on, in a browser. I gave it the "superuser" role. I restarted the server. Not working. Not only that, but trying to log on, in a browser, with this user account + pwd, there is no sign of the server actually being "reached", i.e. no response line in that terminal window saying "user mike12 rejected" (as it was saying earlier with user "elastic"). What could explain that?

Here's what I get when I try to connect using CURL (password is also "mike12"):

D:\...\doc_indexer>curl -k -u mike12:mike12 https://localhost:9500
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [mike12]
for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"",
"Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [mike12]
for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"",
"Bearer realm=\"security\"","ApiKey"]}},"status":401}

... and as I say, no sign in the server terminal that this request has even been "seen".

later
No idea what I did: deleted "mike12" and recreated it. Same user name, same password, same "role" (superuser). Only this time the server was responding, registering user account changes... and then CURL worked... and also browser access. I didn't have to change the "elastic" password. Let's hope "mike12" lasts...

NB the above-referenced page seems a bit out of date for version 8: the names of "roles" used in the examples there don't seem to exist any more. It would help struggling users such as myself to try to keep things up-to-date, as those man pages are the primary source of ES reference!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.