A variety of problems enabling security on elasticsearch

[ire]

I have a version 7.17 of elasticsearch in prod which I need to secure, I have provisioned a similar one with less nodes to test the process but after building some confidence in the process, I am stuck on getting the prod instance enabled, It's one thing or the other.
At the moment, I have all my nodes in prod with

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

in their /etc/elasticsearch/elasticsearch.yml

and when the restart, kibana does not load but show error:

message":"License information could not be obtained from Elasticsearch due to {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_xpack?accept_enterprise=true]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_xpack?accept_enterprise=true]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401} error"

I try to curl the endpoint and I get a similar error

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401

it a prod cluster and it's already taken hours of troubleshooting
Please help urgently, thanks

[ire]

When I tried to revert back and at least get KIbana back up and set

xpack.security.enabled: false
xpack.security.transport.ssl.enabled: false

I restart the nodes and I get the error on

SSL/TLS request received but SSL/TLS is not enabled on this node

so tried to create credentials for the built-in users with

bin/elasticsearch-setup-passwords auto

I got the error

unable to determine default URL from settings, please use the -u option to explicitly provide the url

and when I try with the url I get

SSL connection to https://elasticsearch-in.host:9200/_security/_authenticate?pretty failed: Unsupported or unrecognized SSL message
Please check the elasticsearch SSL settings under xpack.security.http.ssl.

but I did not configure that yet. I only want minimal and basic security

if I use the command without https I get

Failed to authenticate user 'elastic' against http://elasticsearch-in.host:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /etc/elasticsearch/elasticsearch.keystore


ERROR: Failed to verify bootstrap password

I dont have the password for the elastic user

[ire]

then I tried to create a new superuser as advised on

 bin/x-pack/users useradd new_admin -p password -r superuser

but the tool /bin/xpack/users is not available on my version/installation of elasticsearch

now I want to revert all back to disable the security but turning back to false does not have the desired effect as it tells me