Mutual tls/ssl on elasticsearch

Hi,

I want to know how to enable mutual tls/ssl on elastic. I have added the below lines in my elastic.yml file :

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/dev_23_node/dev_23_node.key
xpack.security.http.ssl.certificate: certs/dev_23_node/dev_23_node.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/dev_23_node/dev_23_node.key
xpack.security.transport.ssl.certificate: certs/dev_23_node/dev_23_node.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

If i add above lines in my yml file, is it mutual tls or one way tls enabled. Im little bit confused on this. Please help me on this. Thanks!

Hi there ! Can you clarify exactly what you want to do ? Mutual TLS authentication between which components ? Between elasticsearch nodes themselves? Between Elasticsearch nodes and other components of the Elastic stack ( like Kibana, Logstash, Beats) ? Between Elasticsearch and other clients? Something else entirely ?

Our documentation is quite extensive, I am certain that reading through our docs in Tutorial: Encrypting communications | Elasticsearch Guide [7.3] | Elastic and Encrypting communications in Elasticsearch | Elasticsearch Guide [7.3] | Elastic will be extremely helpful for your understanding.

Hi @ikakavas,

I have gone through the above links already. Mutual SSL authentication is the concept of two parties authenticating each other at the same time. So the client has to provide its own identity to the server and in the same manner server must provide its own identity to the client.

I want to enable mutual tls between elastic cluster and other clients, in my case client is our own standalone developed api. So how to do that ?
Correct me if I am wrong - If i add the above mentioned lines in yml files, i guess it is one way tls.

Please help me on this. Thanks!

See https://www.elastic.co/guide/en/elasticsearch/reference/7.3/security-settings.html#http-tls-ssl-settings

You would need to set xpack.security.http.ssl.client_authentication to required or optional. Then your custom client would need to provide a client certificate when connecting, that is trusted by your certs/ca/ca.crt (as you have set it for the value of xpack.security.http.ssl.certificate_authorities ).

That would take care of the TLS mutual authentication requirement. If you, additionally, want your client to authenticate to Elaticsearch on the application layer with that certificate only you need to set up a PKI realm as we describe in our docs.

Thanks @ikakavas. I have few queries

Q1 : How to generate the client certificate that is trusted bycerts/ca/ca.crt?
Q2 : Right now I am using the below code to connect to elastic cluster, so how to pass client cert in the code ?

String keyStorePath = "C:\\Users\\Documents\\my_keystore_08.jks";
final SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(new File(keyStorePath), trustPass.toCharArray()).build();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("user", "password"));
builder = RestClient.builder(new HttpHost("host", port, "https")).setHttpClientConfigCallback((HttpAsyncClientBuilder httpClientBuilder) -> httpClientBuilder
							.setDefaultCredentialsProvider(credentialsProvider).setSSLContext(sslContext));

Note: I have generated keystore.jks file by passing the ca.crt.

The simplest way I can think of is with elasticsearch-certutil :

bin/elasticsearch-certutil cert --ca-cert certs/ca/ca.crt --ca-key certs/ca/ca.key --pass yourpasswordhere -out client.p12

This PKCS#12 store will now contain both the CA certificate and the client key and certificate, so that you can use it in your client as both the keystore and the truststore

Then you can change the following part:

String trustStorePath = "C:\\Users\\Documents\\my_keystore_08.jks";
final SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(new File(keyStorePath), trustPass.toCharArray()).build();

to

String keyStorePath = "C:\\Users\\Documents\\client.p12";
final SSLContext sslContext = new SSLContextBuilder()
    .loadKeyMaterial(new File(keyStorePath), trustPass.toCharArray())
    .loadTrustMaterial(new File(keyStorePath), trustPass.toCharArray())
    .build();
1 Like

Thansks @ikakavas.

I have generated the client certs by using the above command. I have added all the certs in browser and when I am trying to access elasticsearch through browser its throwing the below error :

 Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:254) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1161) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1152) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        ... 16 more
[2019-09-04T14:28:30,600][INFO ][o.e.c.r.a.AllocationService] [node1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test12][0]] ...]).

Below is my yml config :

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/node1.key
xpack.security.http.ssl.certificate: certs/node1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/node1.key
xpack.security.transport.ssl.certificate: certs/node1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
xpack.security.http.ssl.client_authentication : required 

Please help me on this.

Can anyone please help me on this. Thanks!

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient".

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

Your error message says the client (== your browser) didn't send a certificate and key.

Could we stick to one problem at a time ? Does your Java code work?

You can test your Elasticsearch setup and your generated client.p12 with the following:

curl -k --cert-type P12 --cert client.p12:yourpasswordhere https://<your_host>:9200

Sorry for that @ikakavas.

Now its showing the below error

Caused by: java.security.SignatureException: Signature does not match.
        at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:452) ~[?:?]
        at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166) ~[?:?]
        at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147) ~[?:?]
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:237) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:145) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:344) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:258) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1246) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1171) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1152) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        ... 16 more

When i execute the curl command its showing like below

curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.

When i open the client cert its showing the integrity of this certificate cannot be guaranteed the certificate may be corrupted

I dint try the java code still as im unable to start ES by passing client cert.

So do i need to pass the hostname in the command while generating the client cert. If yes how to pass the host name in the command and generate client cert ?

Please help me on this. Thanks!

You mean, now, as in after what ? I'm trying to understand what you have changed so that you get this error instead of the previous one. And you get it when? When you start Elasticsearch ? When you connect with a client ? What is this client ? a browser ? curl ? your java client ? what is the configuration of that client ?

The more precise you are in describing the behavior and errors you get, the easier will be for someone to help you :slight_smile:

please try by adding -v to curl and share the whole output, not just a line.

What do you "open" it with ? And by the client cert, do you mean client.p12 ?

I'm not sure I understand this. If you can't start Elasticsearch, how are you trying to connect to it as you said above ?

Please take the time to share the current setup in detail and also share at least :

  • Your current elasticsearch.yml
  • How ( with what tools and what commands exactly ) did you generate the certificates and keys you are using there.
  • How exactly did you generate the client.p12
  • Your exact curl command and the associated output

Thanks for the reply @ikakavas.

My current yml file is placed below and i have generated the certs by using instance.yml file using the link - - https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash 2 months back and its working fine with https://node1.elastic.test.com:9200

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/node1.key
xpack.security.http.ssl.certificate: certs/node1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/node1.key
xpack.security.transport.ssl.certificate: certs/node1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
path.repo: C:\Users\suas\Downloads\snapshots
xpack.monitoring.enabled : false
xpack.security.http.ssl.client_authentication : required

curl command to generate clientpem.p12 file - bin\elasticsearch-certutil cert --ca-cert config\certs\ca.crt --ca-key config\certs\node1.key --pass password -out clientpem.p12 and the output is clientpem.p12 cert.

If i disable client authentication its working fine with https but if enable client authentication when am connecting through chrome browser by passing client cert clientpem.p12 im getting the above error which i have mentioned in previous comment.

I have imported the client cert into chrome browser and when i opened in browser its shows error like this - the integrity of this certificate cannot be guaranteed the certificate may be corrupted

Please let me know if you need any more info from my side. Thanks!

Posting the error log again as previous was exceeding the max characters. Thanks!

Caused by: java.security.SignatureException: Signature does not match.
        at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:452) ~[?:?]
        at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166) ~[?:?]
        at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147) ~[?:?]
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:237) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:145) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:344) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:258) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1246) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1171) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1152) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final

This is wrong. You need the CAs key to in order to sign the client certificates, not the node's key. There should be a ca.key file in your certs.zip if you followed the blog you shared with us above. Use that one and your problems will be resolved

Hi @ikakavas,

It will not generate the key for CA, it will key only for key.
You can check the link - https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash

They have mentioned
-rw-r--r--. 1 root elasticsearch 1834 Apr 12 08:47 ca.crt
-rw-r--r--. 1 root elasticsearch 1509 Apr 12 08:47 node1.crt
-rw-r--r--. 1 root elasticsearch 1679 Apr 12 08:47 node1.key

I also have only 3 files mentioned above and there is no key for CA. So how to generate key for CA. Please help me on this. Thanks!

True. That is unfortunate. You can't generate the key for your CA certificate, this is long gone. You would need to recreate your CA certificate and key and also create new node certificate and keys in the process ( that are signed by this new CA ) .

So you need to follow the blog process once again but in step 2-4, instead of running

bin/elasticsearch-certutil cert ca --pem --in ~/tmp/cert_blog/instance.yml --out ~/tmp/cert_blog/certs.zip

you should run

bin/elasticsearch-certutil cert ca --pem --keep-ca-key --in ~/tmp/cert_blog/instance.yml --out ~/tmp/cert_blog/certs.zip

This will ensure that the ca.key will be kept in the ca folder. Once this is done , you can use it to further create new clients certificate as I showed you above

Hope this helps

Hi @ikakavas,

Firstly thanks for helping me.

I tried by above things but it was not working. So i generated new .p12 certs for both client and server and its working fine with client authentication. So i have formed a 3 node elastic cluster by using the same certs for all the 3 nodes. Node-1 and node-2 are working fine. Node-3 is also up and running fine but in console it keeps on throwing the below error :

Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:254) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:377) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]

Below is my yml for node-3:

node.name: node-3
cluster.name: dev_env
node.data: true
node.master: true
network.host: elastic.23.dev
network.publish_host: elastic.23.dev
transport.tcp.port: 9300
transport.publish_port: 9300
discovery.seed_hosts: ["node-1:9300","node-2:9301","node-3:9302"]
discovery.zen.minimum_master_nodes: 1
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate

### p12 Certs ###
xpack.security.transport.ssl.keystore.path: certs/server.p12
xpack.security.transport.ssl.truststore.path: certs/server.p12
xpack.security.http.ssl.keystore.path: certs/server.p12
xpack.security.http.ssl.truststore.path: certs/server.p12
xpack.security.http.ssl.client_authentication : required 

Note: All the nodes are having same certs and same config. Node-1 and node-2 and running fine without any errors in console. Node-3 is running fine but keeps on throwing the above error in console. Please help me on this. Thanks!

It's usually not helpful at all when you just mention something is "not working". No one can help you out as they can't guess what "not working" means, and no one can be helped from reading the forums at a later time as they can't be sure if your problem is applicable to them too.

That's nice, this was the original suggestion too, I just mentioned you should create a PEM CA certificate as it would match the blog post you were following. It shouldn't make any difference.

  • Please share a larger part of your logs, this is not enough.
  • Is something trying to connect to node 3 ? A client of yours ?

Hi @ikakavas,

I have setup 3 node ES cluster with client authentication and all the 3 nodes are working fine but in node-3 console i see below error message coming for every 5 seconds :

[2019-09-09T08:07:05,532][WARN ][o.e.h.AbstractHttpServerTransport] [node-3] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/192.168.0.13:54162}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:254) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:377) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        ... 16 more
[2019-09-09T08:07:05,593][WARN ][o.e.h.AbstractHttpServerTransport] [node-3] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/192.168.0.13:54164}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

No one is acccessing node-3.
Yml file i have shared in my previous reply. Please tell me whats happening here . Thanks!

Your logs beg to disagree.

[2019-09-09T08:07:05,593][WARN ][o.e.h.AbstractHttpServerTransport] [node-3] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/192.168.0.13:54164}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

Something that runs on 192.168.0.13 connects to your node over https on port 9200. In doing so, it doesn't send a client certificate and since client TLS authentication is required ( xpack.security.http.ssl.client_authentication : required) your node fails the request.*
This is expected behavior, you need to figure out what tries to connect to your node 3 and configure it with a client key and certificate.

* The error says "Empty server certificate chain" but this "server" part is misleading, if you read the source code