Strange authentication error

[PTLyon (Pedro Santos)]

Hello, I have been running the ELK stack 8.6.2 for a month now with no problems, but I had security disabled.

Today I attempted to enable security following this documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/security-minimal-setup.html

I have the full ELK stack running on a server and I'm exposing Kibana through nginx. It's working (or was) just fine with 'xpack.security.enabled: false' on the elastic.yml. After following the above documentation in an attempt to setup minimal security, I have encountered an issue.

The issue is that I can't login to either the elastic user, or the kibana_system user through the dashboard. Now, I realize the kibana_system isn't supposed to be able to login on the dashboard and that's fine, but the elastic user can't either which means I can't access the dashboards at all.

The "strange" part about this error is that if I type in the elastic username and a random password, it immediately says password/username incorrect. However, if I type in the correct username and password, it appears to login and start loading for a moment, before I'm brought back to this screen:

I actually get to see this:

But then it just brings me back to the login error screen.

I apologize for any information I might not have given right off the bat, I'm fairly new to the ELK Stack. If there's any more information I can provide to help you help me, please do let me know.
I tried to find out how to read kibana logs so I could figure out what was happening but everything I found was through the dashboard, which I can't access at the moment.

Any help would be very much appreciated. Thank you.

[leandrojmp (Leandro Pereira)]

@PTLyon

I tried to find out how to read kibana logs so I could figure out what was happening but everything I found was through the dashboard, which I can't access at the moment.

Can you share the logs of both Kibana and Elasticsearch? Share the logs of the Elasticsearch instance configured in your kibana.yml.

[PTLyon (Pedro Santos)]

Yes! I was actually about to edit to add logs.

var/log/kibana generates the following log entry when I try to authenticate:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-03-23T19:38:32.317+00:00","message":"Authentication attempt failed: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [kibanaorwell] for REST request [/_security/_authenticate]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [kibanaorwell] for REST request [/_security/_authenticate]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","log":{"level":"INFO","logger":"plugins.security.authentication"},"process":{"pid":5697},"trace":{"id":"58d53d53d4982eccbeb2a5dbeff5f5b9"},"transaction":{"id":"5b441aaed56ddd01"}}

This is strange because kibanaorwell is the username I have for Nginx before I had security enabled.

The following is the Elasticsearch log entry from when I tried to login:

[2023-03-23T19:31:54,179][INFO ][o.e.x.s.a.RealmsAuthenticator] [bigbrother] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

I found this thread a couple of minutes ago and I think I might be suffering from a similar problem.

I apologize for not mentioning I had been using Nginx before I enabled security, I had no idea it could cause this sort of issue.

I am using Nginx to be able to access the Kibana dashboard from my ELK server's public IP address. This was a solution suggested to me at the time when I started with 7.x.
Is there no longer a need for this or do I still need to use Nginx to do it but have to change configurations?

Thank you for taking the time to reply. If there's anything else I can do or if I need to provide more logs just let me know.