# Mutual tls/ssl on elasticsearch

**URL:** <https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [September 3, 2019, 6:57am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768 "2019-09-03T06:57:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 3, 2019, 6:57am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/1 "2019-09-03T06:57:14Z")

</div>

Hi,

I want to know how to enable mutual tls/ssl on elastic. I have added the below lines in my elastic.yml file :

```
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/dev_23_node/dev_23_node.key
xpack.security.http.ssl.certificate: certs/dev_23_node/dev_23_node.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/dev_23_node/dev_23_node.key
xpack.security.transport.ssl.certificate: certs/dev_23_node/dev_23_node.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

```

If i add above lines in my yml file, is it mutual tls or one way tls enabled. Im little bit confused on this. Please help me on this. Thanks!

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 3, 2019, 7:19am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/2 "2019-09-03T07:19:06Z")

</div>

> [@SukeshGupta](#):
>
> I want to know how to enable mutual tls/ssl on elastic.

Hi there ! Can you clarify exactly what you want to do ? Mutual TLS authentication between which components ? Between elasticsearch nodes themselves? Between Elasticsearch nodes and other components of the Elastic stack ( like Kibana, Logstash, Beats) ? Between Elasticsearch and other clients? Something else entirely ?

Our documentation is quite extensive, I am certain that reading through our docs in [Tutorial: Encrypting communications | Elasticsearch Guide [7.3] | Elastic](https://www.elastic.co/guide/en/elastic-stack-overview/7.3/encrypting-internode-communications.html) and [Encrypting communications in Elasticsearch | Elasticsearch Guide [7.3] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/configuring-tls.html) will be extremely helpful for your understanding.

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 3, 2019, 7:36am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/3 "2019-09-03T07:36:15Z")

</div>

Hi @ikakavas,

I have gone through the above links already. **Mutual SSL authentication is the concept of two parties authenticating each other at the same time. So the client has to provide its own identity to the server and in the same manner server must provide its own identity to the client.**

I want to enable mutual tls between elastic cluster and other clients, in my case client is our own standalone developed api. So how to do that ?  
Correct me if I am wrong - If i add the above mentioned lines in yml files, i guess it is one way tls.

Please help me on this. Thanks!

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 3, 2019, 7:52am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/4 "2019-09-03T07:52:23Z")

</div>

See [https://www.elastic.co/guide/en/elasticsearch/reference/7.3/security-settings.html#http-tls-ssl-settings](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/security-settings.html#http-tls-ssl-settings)

You would need to set `xpack.security.http.ssl.client_authentication` to `required` or `optional`. Then your custom client would need to provide a client certificate when connecting, that is trusted by your `certs/ca/ca.crt` (as you have set it for the value of `xpack.security.http.ssl.certificate_authorities` ).

That would take care of the TLS mutual authentication requirement. If you, additionally, want your client to authenticate to Elaticsearch on the application layer with that certificate only you need to set up a [PKI realm](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/configuring-pki-realm.html) as we describe in our docs.

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 3, 2019, 8:55am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/5 "2019-09-03T08:55:10Z")

</div>

Thanks @ikakavas. I have few queries

Q1 : How to generate the client certificate that is trusted by`certs/ca/ca.crt`?  
Q2 : Right now I am using the below code to connect to elastic cluster, so how to pass client cert in the code ?

```
String keyStorePath = "C:\\Users\\Documents\\my_keystore_08.jks";
final SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(new File(keyStorePath), trustPass.toCharArray()).build();
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("user", "password"));
builder = RestClient.builder(new HttpHost("host", port, "https")).setHttpClientConfigCallback((HttpAsyncClientBuilder httpClientBuilder) -> httpClientBuilder
							.setDefaultCredentialsProvider(credentialsProvider).setSSLContext(sslContext));

```

**Note: I have generated keystore.jks file by passing the ca.crt.**

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 4, 2019, 8:24am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/6 "2019-09-04T08:24:26Z")

</div>

> [@SukeshGupta](#):
>
> Q1 : How to generate the client certificate that is trusted by `certs/ca/ca.crt` ?

The simplest way I can think of is with [`elasticsearch-certutil`](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html) :

```auto
bin/elasticsearch-certutil cert --ca-cert certs/ca/ca.crt --ca-key certs/ca/ca.key --pass yourpasswordhere -out client.p12

```

> [@SukeshGupta](#):
>
> Q2 : Right now I am using the below code to connect to elastic cluster, so how to pass client cert in the code ?

This `PKCS#12` store will now contain both the CA certificate and the client key and certificate, so that you can use it in your client as both the keystore and the truststore

Then you can change the following part:

```auto
String trustStorePath = "C:\\Users\\Documents\\my_keystore_08.jks";
final SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(new File(keyStorePath), trustPass.toCharArray()).build();

```

to

```auto
String keyStorePath = "C:\\Users\\Documents\\client.p12";
final SSLContext sslContext = new SSLContextBuilder()
    .loadKeyMaterial(new File(keyStorePath), trustPass.toCharArray())
    .loadTrustMaterial(new File(keyStorePath), trustPass.toCharArray())
    .build();

```

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 4, 2019, 9:02am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/7 "2019-09-04T09:02:23Z")

</div>

Thansks @ikakavas.

I have generated the client certs by using the above command. I have added all the certs in browser and when I am trying to access elasticsearch through browser its throwing the below error :

```
 Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:254) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1161) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1152) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        ... 16 more
[2019-09-04T14:28:30,600][INFO][o.e.c.r.a.AllocationService] [node1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test12][0]] ...]).

```

Below is my yml config :

```
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/node1.key
xpack.security.http.ssl.certificate: certs/node1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/node1.key
xpack.security.transport.ssl.certificate: certs/node1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
xpack.security.http.ssl.client_authentication : required 

```

Please help me on this.

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 5, 2019, 10:35am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/8 "2019-09-05T10:35:25Z")

</div>

Can anyone please help me on this. Thanks!

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 5, 2019, 2:09pm UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/9 "2019-09-05T14:09:12Z")

</div>

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the [Community Code of Conduct](https://www.elastic.co/community/codeofconduct) specifically the section "Be patient".

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a [subscription](https://www.elastic.co/subscriptions).

Your error message says the client (== your browser) didn't send a certificate and key.

Could we stick to one problem at a time ? Does your Java code work?

You can test your Elasticsearch setup and your generated `client.p12` with the following:

```auto
curl -k --cert-type P12 --cert client.p12:yourpasswordhere https://<your_host>:9200

```

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 6, 2019, 5:51am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/10 "2019-09-06T05:51:02Z")

</div>

Sorry for that @ikakavas.

Now its showing the below error

```
Caused by: java.security.SignatureException: Signature does not match.
        at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:452) ~[?:?]
        at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166) ~[?:?]
        at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147) ~[?:?]
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:237) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:145) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:344) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:258) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1246) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1171) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1152) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        ... 16 more

```

When i execute the curl command its showing like below

`curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.`

When i open the client cert its showing `the integrity of this certificate cannot be guaranteed the certificate may be corrupted`

**I dint try the java code still as im unable to start ES by passing client cert.**

So do i need to pass the hostname in the command while generating the client cert. If yes how to pass the host name in the command and generate client cert ?

Please help me on this. Thanks!

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 6, 2019, 6:28am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/11 "2019-09-06T06:28:55Z")

</div>

> [@SukeshGupta](#):
>
> Now its showing the below error

You mean, now, as in after what ? I'm trying to understand what you have changed so that you get this error instead of the previous one. And you get it when? When you start Elasticsearch ? When you connect with a client ? What is this client ? a browser ? curl ? your java client ? what is the configuration of that client ?

The more precise you are in describing the behavior and errors you get, the easier will be for someone to help you 🙂

> [@SukeshGupta](#):
>
> When i execute the curl command its showing like below
> 
> `curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.`

please try by adding `-v` to curl and share the whole output, not just a line.

> [@SukeshGupta](#):
>
> When i open the client cert its showing `the integrity of this certificate cannot be guaranteed the certificate may be corrupted`

What do you "open" it with ? And by the client cert, do you mean `client.p12` ?

> [@SukeshGupta](#):
>
> **I dint try the java code still as im unable to start ES by passing client cert.**

I'm not sure I understand this. If you can't start Elasticsearch, how are you trying to connect to it as you said above ?

Please take the time to share the current setup in detail and also share at least :

- Your current elasticsearch.yml
- How ( with what tools and what commands _exactly_ ) did you generate the certificates and keys you are using there.
- How _exactly_ did you generate the client.p12
- Your _exact_ curl command and the associated output

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 6, 2019, 6:39am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/12 "2019-09-06T06:39:09Z")

</div>

Thanks for the reply @ikakavas.

My current yml file is placed below and i have generated the certs by using instance.yml file using the link - - [https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash](https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash) 2 months back and its working fine with **[https://node1.elastic.test.com:9200](https://node1.elastic.test.com:9200)**

```
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/node1.key
xpack.security.http.ssl.certificate: certs/node1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/node1.key
xpack.security.transport.ssl.certificate: certs/node1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
path.repo: C:\Users\suas\Downloads\snapshots
xpack.monitoring.enabled : false
xpack.security.http.ssl.client_authentication : required

```

curl command to generate clientpem.p12 file - `bin\elasticsearch-certutil cert --ca-cert config\certs\ca.crt --ca-key config\certs\node1.key --pass password -out clientpem.p12` and the output is **clientpem.p12** cert.

If i disable client authentication its working fine with https but if enable client authentication when am connecting through chrome browser by passing client cert **clientpem.p12** im getting the above error which i have mentioned in previous comment.

I have imported the client cert into chrome browser and when i opened in browser its shows error like this - **the integrity of this certificate cannot be guaranteed the certificate may be corrupted**

Please let me know if you need any more info from my side. Thanks!

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 6, 2019, 8:16am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/13 "2019-09-06T08:16:12Z")

</div>

Posting the error log again as previous was exceeding the max characters. Thanks!

```
Caused by: java.security.SignatureException: Signature does not match.
        at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:452) ~[?:?]
        at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166) ~[?:?]
        at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147) ~[?:?]
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:237) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:145) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:344) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:258) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1246) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1171) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1152) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final
```

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 6, 2019, 10:21am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/14 "2019-09-06T10:21:34Z")

</div>

> [@SukeshGupta](#):
>
> curl command to generate clientpem.p12 file - `bin\elasticsearch-certutil cert --ca-cert config\certs\ca.crt --ca-key config\certs\node1.key --pass password -out clientpem.p12` and the output is **clientpem.p12** cert.

> [@SukeshGupta](#):
>
> --ca-key config\certs\node1.key

This is wrong. You need the CAs key to in order to sign the client certificates, not the node's key. There should be a `ca.key` file in your `certs.zip` if you followed the blog you shared with us above. Use that one and your problems will be resolved

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 6, 2019, 10:48am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/15 "2019-09-06T10:48:33Z")

</div>

Hi @ikakavas,

It will not generate the key for CA, it will key only for key.  
You can check the link - **[https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash](https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash)**

They have mentioned  
-rw-r--r--. 1 root elasticsearch 1834 Apr 12 08:47 ca.crt  
-rw-r--r--. 1 root elasticsearch 1509 Apr 12 08:47 node1.crt  
-rw-r--r--. 1 root elasticsearch 1679 Apr 12 08:47 node1.key

I also have only 3 files mentioned above and there is no key for CA. So how to generate key for CA. Please help me on this. Thanks!

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 6, 2019, 11:10am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/16 "2019-09-06T11:10:19Z")

</div>

> [@SukeshGupta](#):
>
> It will not generate the key for CA,

True. That is unfortunate. You can't generate the key for your CA certificate, this is long gone. You would need to recreate your CA certificate and key and also create new node certificate and keys in the process ( that are signed by this new CA ) .

So you need to follow the blog process once again but in step 2-4, instead of running

```auto
bin/elasticsearch-certutil cert ca --pem --in ~/tmp/cert_blog/instance.yml --out ~/tmp/cert_blog/certs.zip

```

you should run

```auto
bin/elasticsearch-certutil cert ca --pem --keep-ca-key --in ~/tmp/cert_blog/instance.yml --out ~/tmp/cert_blog/certs.zip

```

This will ensure that the `ca.key` will be kept in the `ca` folder. Once this is done , you can use it to further create new clients certificate as I showed you above

Hope this helps

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 9, 2019, 7:25am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/17 "2019-09-09T07:25:10Z")

</div>

Hi @ikakavas,

Firstly thanks for helping me.

I tried by above things but it was not working. So i generated new .p12 certs for both client and server and its working fine with client authentication. So i have formed a 3 node elastic cluster by using the same certs for all the 3 nodes. Node-1 and node-2 are working fine. Node-3 is also up and running fine but in console it keeps on throwing the below error :

```
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:254) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:377) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]

```

Below is my yml for node-3:

```
node.name: node-3
cluster.name: dev_env
node.data: true
node.master: true
network.host: elastic.23.dev
network.publish_host: elastic.23.dev
transport.tcp.port: 9300
transport.publish_port: 9300
discovery.seed_hosts: ["node-1:9300","node-2:9301","node-3:9302"]
discovery.zen.minimum_master_nodes: 1
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate

### p12 Certs ###
xpack.security.transport.ssl.keystore.path: certs/server.p12
xpack.security.transport.ssl.truststore.path: certs/server.p12
xpack.security.http.ssl.keystore.path: certs/server.p12
xpack.security.http.ssl.truststore.path: certs/server.p12
xpack.security.http.ssl.client_authentication : required 

```

Note: All the nodes are having same certs and same config. Node-1 and node-2 and running fine without any errors in console. Node-3 is running fine but keeps on throwing the above error in console. Please help me on this. Thanks!

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 9, 2019, 7:54am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/18 "2019-09-09T07:54:49Z")

</div>

> [@SukeshGupta](#):
>
> I tried by above things but it was not working.

It's usually not helpful at all when you just mention something is "not working". No one can help you out as they can't guess what "not working" means, and no one can be helped from reading the forums at a later time as they can't be sure if your problem is applicable to them too.

> [@SukeshGupta](#):
>
> So i generated new .p12 certs for both client and server and its working fine with client authentication.

That's nice, this was the original suggestion too, I just mentioned you should create a PEM CA certificate as it would match the blog post you were following. It shouldn't make any difference.

> [@SukeshGupta](#):
>
> Note: All the nodes are having same certs and same config. Node-1 and node-2 and running fine without any errors in console. Node-3 is running fine but keeps on throwing the above error in console. Please help me on this. Thanks!

- Please share a larger part of your logs, this is not enough.
- Is something trying to connect to node 3 ? A client of yours ?

---

<div class="post-metadata">

**Author:** ![SukeshGupta](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@SukeshGupta](https://discuss.elastic.co/u/SukeshGupta)\
**Post date:** [September 9, 2019, 8:21am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/19 "2019-09-09T08:21:40Z")

</div>

Hi @ikakavas,

I have setup 3 node ES cluster with client authentication and all the 3 nodes are working fine but in node-3 console i see below error message coming for every 5 seconds :

```
[2019-09-09T08:07:05,532][WARN][o.e.h.AbstractHttpServerTransport] [node-3] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/192.168.0.13:54162}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:254) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:377) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:441) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:688) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]
        ... 16 more
[2019-09-09T08:07:05,593][WARN][o.e.h.AbstractHttpServerTransport] [node-3] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/192.168.0.13:54164}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

```

**No one is acccessing node-3.**  
Yml file i have shared in my previous reply. Please tell me whats happening here . Thanks!

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 9, 2019, 10:58am UTC](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768/20 "2019-09-09T10:58:34Z")

</div>

> [@SukeshGupta](#):
>
> **No one is acccessing node-3.**

Your logs beg to disagree.

```auto
[2019-09-09T08:07:05,593][WARN][o.e.h.AbstractHttpServerTransport] [node-3] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/192.168.0.13:54164}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

```

Something that runs on `192.168.0.13` connects to your node over https on port 9200. In doing so, it doesn't send a client certificate and since client TLS authentication is required ( `xpack.security.http.ssl.client_authentication : required`) your node fails the request.\*  
This is expected behavior, you need to figure out what tries to connect to your node 3 and configure it with a client key and certificate.

\* The error says "Empty server certificate chain" but this "server" part is misleading, [if you read the source code](https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/sun/security/ssl/CertificateMessage.java#L371)

[Next page](https://discuss.elastic.co/t/mutual-tls-ssl-on-elasticsearch/197768.md?page=2)