Problem attempting to migrate from self-signed SSL cert to a CA issued SSL cert

[Darrell_Lowrance]

Elasticsearch version: 6.1.2
3 node cluster (Ubuntu 16.04)

I've installed X-Pack and created a self-signed certificate following the steps at https://www.elastic.co/blog/elasticsearch-security-configure-tls-ssl-pki-authentication

I first replaced only the http ssl cert. Everything works fine. See elasticsearch.yml settings below.

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/mydomaincert.p12
xpack.security.http.ssl.keystore.password: fakepass0rd
xpack.security.http.ssl.truststore.path: certs/mydomaincert.p12
xpack.security.http.ssl.truststore.password: fakepass0rd

xpack.security.http.ssl.client_authentication: optional

Next, I attempted to use CA issued mydomaincert.p12 for the Transport layer.

• It's a wildcard cert, e.g, *.mydomaincert.com. All nodes are on this domain.
• The cert includes Extended Key Usage for both Server and Client Auth.
• I chained together the 3 intermediate certs provide by the CA and included them in the .p12 file.

Just like with the http settings, I updated all ES nodes transport settings to use our .p12 See below.

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/mydomaincert.p12
xpack.security.transport.ssl.keystore.password: fakepass0rd
xpack.security.transport.ssl.truststore.path: certs/mydomaincert.p12
xpack.security.transport.ssl.truststore.password: fakepass0rd

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/mydomaincert.p12
xpack.security.http.ssl.keystore.password: fakepass0rd
xpack.security.http.ssl.truststore.path: certs/mydomaincert.p12
xpack.security.http.ssl.truststore.password: fakepass0rd

xpack.security.http.ssl.client_authentication: optional

I'm able to restart the ES service on all nodes, but I'm no longer able to authenticate to https://mynode:9200. I checked the cluster logs and found these errors-- null cert chain, Received fatal alert: bad_certificate

Log output--

[2019-08-15T10:16:32,286][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [elk-node02] exception caught on transport layer [org.elasticsearch.transport.netty4.NettyTcpChannel@64606c80], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate

And…

Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:306) ~[?:?]
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1939) ~[?:?]

Is there anything I'm missing or should check? I've already spent a good deal of time researching. I've done my best to confirm that all the certs are chained in the .p12 file.

Thanks in advance.

[ikakavas (Ioannis Kakavas)]

It sounds like your PKCS#12 is only containing CA certificates ? In the transport layer, each node is both a client and a server in the context of a TLS connection and as such each node needs to have a 'Private Key / Certificate' pair that is contained in their Keystore that it can use as a client.

Can you share the output of

openssl pkcs12 -in certs/mydomaincert.p12 -info -noout

so that we can verify this line of thinking ?

You would need to generate a key/certificate for each node, with the certificate being signed by the org CA so that the other nodes can trust it.

Please keep in mind that if this CA is used for other things in your organization, configuring it to be the source of trust information for the transport layer is not a very good idea. The reason is that anyone who can get their hands on a certificate signed by that CA, can impersonate an Elasticsearch node and join the cluster. It would be much more secure/prudent to keep using the self-signed certificates for transport layer, in that case.

[Darrell_Lowrance]

Thanks for your quick response.

See out from the openssl command below.

openssl pkcs12 -in ct.p12 -info -noout

Enter Import Password:
MAC Iteration 2048
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Certificate bag
Certificate bag
Certificate bag
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Regarding your comment--

It would be much more secure/prudent to keep using the self-signed certificates for transport layer, in that case.

I've been looking into our application prod environment more. At this particular time, all resources (both server/client for Transport and HTTP connections) will be on private subnets only. So it sounds like using the self-signed cert for all connections is fine, and even desired, for all ELK communications in this closed environment. We'll be effectively acting as a Private CA.

Any other advice would be appreciated.

It looks like the self cert has an expiration date, so we'll need to monitor and create/deploy a new one when necessary, correct?

Thanks!