A variety of problems enabling security on elasticsearch

I have a version 7.17 of elasticsearch in prod which I need to secure, I have provisioned a similar one with less nodes to test the process but after building some confidence in the process, I am stuck on getting the prod instance enabled, It's one thing or the other.
At the moment, I have all my nodes in prod with

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

in their /etc/elasticsearch/elasticsearch.yml

and when the restart, kibana does not load but show error:

message":"License information could not be obtained from Elasticsearch due to {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_xpack?accept_enterprise=true]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_xpack?accept_enterprise=true]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401} error"

I try to curl the endpoint and I get a similar error

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401

it a prod cluster and it's already taken hours of troubleshooting
Please help urgently, thanks

When I tried to revert back and at least get KIbana back up and set

xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false

I restart the nodes and I get the error on

SSL/TLS request received but SSL/TLS is not enabled on this node

so tried to create credentials for the built-in users with

bin/elasticsearch-setup-passwords auto

I got the error

unable to determine default URL from settings, please use the -u option to explicitly provide the url

and when I try with the url I get

SSL connection to https://elasticsearch-in.host:9200/_security/_authenticate?pretty failed: Unsupported or unrecognized SSL message
Please check the elasticsearch SSL settings under xpack.security.http.ssl.

but I did not configure that yet. I only want minimal and basic security

if I use the command without https I get

Failed to authenticate user 'elastic' against http://elasticsearch-in.host:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /etc/elasticsearch/elasticsearch.keystore


ERROR: Failed to verify bootstrap password

I dont have the password for the elastic user

then I tried to create a new superuser as advised on

 bin/x-pack/users useradd new_admin -p password -r superuser

but the tool /bin/xpack/users is not available on my version/installation of elasticsearch

now I want to revert all back to disable the security but turning back to false does not have the desired effect as it tells me

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.