Problem attempting to migrate from self-signed SSL cert to a CA issued SSL cert

Elastic Stack Elasticsearch
1

Elasticsearch version: 6.1.2
3 node cluster (Ubuntu 16.04)

I've installed X-Pack and created a self-signed certificate following the steps at https://www.elastic.co/blog/elasticsearch-security-configure-tls-ssl-pki-authentication

I first replaced only the http ssl cert. Everything works fine. See elasticsearch.yml settings below.

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/mydomaincert.p12
xpack.security.http.ssl.keystore.password: fakepass0rd
xpack.security.http.ssl.truststore.path: certs/mydomaincert.p12
xpack.security.http.ssl.truststore.password: fakepass0rd

xpack.security.http.ssl.client_authentication: optional

Next, I attempted to use CA issued mydomaincert.p12 for the Transport layer.

  • It's a wildcard cert, e.g, *.mydomaincert.com. All nodes are on this domain.
  • The cert includes Extended Key Usage for both Server and Client Auth.
  • I chained together the 3 intermediate certs provide by the CA and included them in the .p12 file.

Just like with the http settings, I updated all ES nodes transport settings to use our .p12 See below.

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/mydomaincert.p12
xpack.security.transport.ssl.keystore.password: fakepass0rd
xpack.security.transport.ssl.truststore.path: certs/mydomaincert.p12
xpack.security.transport.ssl.truststore.password: fakepass0rd

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/mydomaincert.p12
xpack.security.http.ssl.keystore.password: fakepass0rd
xpack.security.http.ssl.truststore.path: certs/mydomaincert.p12
xpack.security.http.ssl.truststore.password: fakepass0rd

xpack.security.http.ssl.client_authentication: optional

I'm able to restart the ES service on all nodes, but I'm no longer able to authenticate to https://mynode:9200. I checked the cluster logs and found these errors-- null cert chain, Received fatal alert: bad_certificate

Log output--

[2019-08-15T10:16:32,286][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [elk-node02] exception caught on transport layer [org.elasticsearch.transport.netty4.NettyTcpChannel@64606c80], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate

And…

Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:306) ~[?:?]
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1939) ~[?:?]

Is there anything I'm missing or should check? I've already spent a good deal of time researching. I've done my best to confirm that all the certs are chained in the .p12 file.

Thanks in advance.