After having xpack security in place in elasticsearch cluster, cluster not working as expected

[surender (surender reddy)]

Initially deployed elasticsearch cluster with 3 nodes and it was started properly and working.

I want to encrypt communication between nodes and between clients and clusters.

configured TLS and http encrypted in the cluster then cluster not working as expected. getting below errors. Please check and help.

below are the entries from elasticsearch.yml file using

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/cert.p12
xpack.security.transport.ssl.truststore.path: certs/cert.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/cert.p12
xpack.security.http.ssl.truststore.path: certs/cert.p12
xpack.security.http.ssl.client_authentication: optional
xpack.security.authc.realms.pki1.type: pki
xpack.security.authc.accept_default_password: false

error from journal log:

polkitd[567]: Unregistered Authentication Agent for unix-processl log :

from elasticcluster log :

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f636c75737465722f73746174653f70726574747920485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a2031302e3132392e3139352e3134393a393230300d0a4163636570743a202a

[ikakavas (Ioannis Kakavas)]

Hi there,

then cluster not working as expected

You need to give us more details. What doesn't work as expected ?

below are the entries from elasticsearch.yml file using

You have also enabled the PKI realm for authentication. This means that every request to Elasticsearch on the http layer needs to be over TLS with client authentication. Is that what you want to achieve? Our PKI realm docs are here : PKI user authentication | Elastic Stack Overview [7.4] | Elastic

from elasticcluster log :

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f636c75737465722f73746174653f70726574747920485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a2031302e3132392e3139352e3134393a393230300d0a4163636570743a202a

This is not the cause of the problem. What you are seeing there is Elasticsearch logging that while it is configured to only accept connections on the http layer over TLS ( you have configured it as such with xpack.security.http.ssl.enabled: true ), it received a connection over plain http. In fact that long HEX string decodes to :

GET /_cluster/state?pretty HTTP/1.1
User-Agent: curl/7.29.0
Host: 10.129.195.149:9200
Accept: *