TSL Transport layer "valid certification path to requested target"

[jordo1138]

There are many topics around transport layer tls and fixes etc but the issue I"m running into is on a 3 node cluster ES 6.3 where I have placed the my root ca cert, wild card cert and key in to the /etc/elasticsearch folder on ubuntu 16 and when the transport validation mode is "certificate"
i get the error below.
when i curl each node for _xpack/certificates i get the expected result also with the listings of my cert information. When i keep the verification_mode: none of course the 3 node cluster is up and communicating but i'm stumped as to why the clients don't trust the cert when the mode is certificate level
and as far as i understand when using pem format, the path to the ca cert authorities as shown in the yml excerpt below should be good to go
any ideas?
Thanks,

My elasticsearch.yml xpack settings are here:
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/devops.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/devops.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/xxxxxxroot.crt" ]

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_171]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
at

[TimV (Tim Vernum)]

What does the certificate chain look like from /etc/elasticsearch/xxxxxxroot.crt to /etc/elasticsearch/devops.crt

Does the root CA sign it directly, or is it signed by an intermediate CA?
Is what you've listed the CA really a root CA (that is, self signed)?

If you have access to an openssl binary, can you try:

openssl verify -verbose -CAfile /etc/elasticsearch/xxxxxxroot.crt /etc/elasticsearch/devops.crt

The error messages that openssl verify gives can be a bit cryptic, but it will at least check whether the trust chain is correct.

[jordo1138]

Ok, so probly part of the problem... the openssl verify gave out the error below
"error 20 at 0 depth lookup:unable to get local issuer certificate"

I did a look into it, and it is the CA issued by Comodo where as the devops.crt is by digicert...they were handed to me to use but there could be a breakdown somewhere...
I may go back and just use ES6.3 utilities to generate a new set just for the transport layer but my goal was to use our wildcard cert (devops.crt) so that the 3 node hostnames could be *.devops.X.com
Thanks,

[TimV (Tim Vernum)]

@jordo1138

there could be a breakdown somewhere

It certainly looks that way.

You've got a few options here:

1. You can just pull down the CA that signed the devops certificate. If it's from digicert then you should be able to easily get the CA and point to it.
   However, that does mean that anyone with a digicert certificate can estalbish their identity and be trusted by your cluster. When you configure xpack.security.transport.ssl.certificate_authorities you are effectively stating that you are willing to trust all certificates signed by those CAs, and let them join your cluster. You might have added protections at a network level, but the TLS based trust isn't going to help you there.

2. Generate a new CA of your own, and use that to create certificates for the transport layer on your nodes. If you do that, and keep all the keys locked-down & private, then you can generally turn off hostname verification, and you don't need to worry about having a wildcard cert. So it's possible to use a single cert (without any hostname information) for all the nodes. All you could have 1 cert that includes the hostname for all 3 nodes.

The trust model for a closed cluster with its own private CA is quite different to the trust model for public websites. For public sites, where it is expected (and largely required) that every site on the internet has access to some certificate from one of the CAs that you trust, then you can't simply rely on that fact that a website has a certificate to establish any sort of trust - because effectively everyone has a certificate. So, in that case hostname verification is adding a required check of "Are you who I think you are?"
But if you have your own private CA, and you are very strict about who you give certificates to (that is, you only give certificates to nodes that are supposed to be in your cluster) and you do a proper job of protecting the private keys for the CA and the node certificates, then the fact that a node has access to a certificate (from your CA) is all you need - that is sufficient to tell you that they are supposed to be part of your cluster. In that case hostname verification adds some minor incremental value in that if someone steals the private key for your cert, then they would also have to do some sort of hostname spoofing to be able to use it, but you need to decide whether that benefit is worth the pain of managing hostnames for your certs and regenerating them if you change hosts.

[jordo1138]

Thanks, I'll probably try the ES utilities for generating new CA and certs