Has anyone actually gotten a cert to install? I always get "non-null policy tree required and policy tree is null"
I have entered the docker container and replaced manually the 2 pems used by stunnel, but don't know where I'd manually place them to make them stick.
I assume that error you are getting is when uploading from the API or UI?
This post has proven useful in the past in diagnosing why a cert has not been constructed "correctly" (where "correctly" means whatever Java library we use accepts it!): ECE 2.0.1 Unable to upload Letsencrypt TLS certificate
Alex
Yes I have tried from UI and curl.
It could be Java doesn't trust the root. It's an enterprise root. Is there a way to import Java cacerts into ECE? I also tried manually editing the Java cacerts but it may have not kept between service restarts (also not sure which container does this check)
Unfortunately there is currently no (persistent) way to edit the docker container's trusted CA list - to use an internal CA I believe it's necessary to terminate it at the load balancer that sits in front of the ECE proxies. (and then use the self-signed ECE certs, or external certs with a trusted CA, for the final hop).
(I actually have no idea what would happen if you hand edited the frc-admin-consoles-admin-console services' cacerts for long enough to upload the certs, which is the only time we check their validity, it's not something we've ever tried)
It's probably best I do setup a proxy now just to keep what I can behind port 443 for future ease with customer firewalls...
but I'm stubborn and looked at this at home using Let's Encrypt's root Fake LE Root X1 from the staging environment which obviously isn't in the truststore.
I entered the frc-admin-consoles-admin-console and added it like so:
[root@scelastic ~]# docker exec -it frc-admin-consoles-admin-console bash
[root@cec4e0674b75 /]# cd /etc/pki/ca-trust/source/anchors
[root@cec4e0674b75 anchors]# cat >fakelerootx1.pem
<paste cert in>
[root@cec4e0674b75 anchors]# update-ca-trust
[root@cec4e0674b75 anchors]# pkill java
That worked. Survived a reboot anyway.
It may cause trouble if other services need to trust the UI / Proxy cert, but the ece ca master wasn't in cacerts to begin with... So while I won't use this in production I may keep it in my lab to see what breaks.
That was not the problem with our enterprise cert.
[elastic@scelastic ~]$ curl -k -H Expect: -H 'Content-Type: application/json' --data-binary @./test.pem --user "admin:$pass" https://localhost:12443/api/v1/platform/configuration/security/tls/ui
{
"errors": [{
"code": "cert.invalid_cert_chain",
"message": "Certificate chain was invalid [non-null policy tree required and policy tree is null]"
}]
}
[elastic@scelastic ~]$ curl -k -H Expect: -H 'Content-Type: application/json' --data-binary @./certbot/live/scelastic.domain/keychain.pem --user "admin:$pass" https://localhost:12443/api/v1/platform/configuration/security/tls/ui
{
}
But I have a proxy in front and get to also map the cluster UUIDs to meaningful names so at least there is that...
I had a look through our support database to see if anyone else had ever reported the specific non-null policy tree error and I found one hit, where (after lots of dead ends and red herrings):
[REDACTED] ultimately had to create their own certs because the intermediate they were using didn't comply with section 6.1.3 (section f) of RFC5280
which (fwiw!) says:
(f) Verify that either explicit_policy is greater than 0 or the
valid_policy_tree is not equal to NULL;
(For completeness, I also found one hit on the entire internet (!! well actually 2 but the other one was you 
): SSLHandshakeException - java.security.cert.CertPathValidatorException: non-null policy tree required and policy tree is null - Stack Overflow, but alas the post never got answered)
Thank you for continuing to look at this. Since I'm the only one in the world with the problem I'll continue documenting what I found to avoid a xkcd: Wisdom of the Ancients situation. 
I set -Djava.security.debug=all in my /service/adminconsole/run java command line.
Here is final page of side-by-side diff of my Fake LE cert on the left and a DOD cert on the right.
The application doesn't care about a policy so validator should be checking for anyPolicy. But they aren't ending up in valid_policy_three and throws an exception in PolicyChecker.processPolicies() no matter if initPolicies was anyPolicy:
511 // PKIX: Section 6.1.3: Step (f)
512 // verify that either explicit policy is greater than 0 or
513 // the valid_policy_tree is not equal to NULL
514 if ((explicitPolicy == 0) && (rootNode == null)) {
515 throw new CertPathValidatorException
516 ("non-null policy tree required and policy tree is null",
517 null, null, -1, PKIXReason.INVALID_POLICY);
518 }
A similar issue with certpath verification was found in NSS by Chrome:
Reported upstream:
I feel like it should not do that if initialPolicies was empty. The Chrome workaround was to go back through and check it for the policy on the leaf certificate. I say this is a Java bug in sun.security.provider.certpath.PolicyChecker
After reviewing how SSLSockets validate certs, I think ECE could use a simplier validator and is likely the best solution to this problem. I probably will have to wait until our support contract starts to open the bug report? Very new to elastic (and java)..
Looks like eventually no.found.security.certificate.CertChain.apply calls the PKIX validator when Validator.VAR_TLS_SERVER is probably more appropriate.
I opened an issue in our developer repo to investigate alternative validators, for our API team, but if/when you do have a support contract, please do raise an issue since we use those to help with our prioritization
Alex
After future digging and hacking away at the OpenJDK classes ... I found this is not a bug in Java. Further inspection of RFC shows we should start path checks at the intermediate and things in the trust anchor needn't be checked.
When the trust anchor is provided in the form of a self-signed certificate, this self-signed certificate is not included as part of the prospective certification path. Information about trust anchors is provided as inputs to the certification path validation algorithm (Section 6.1.1).
So all that needs done is to remove the ca from the chain before calling generateCertPath()
I crafted a test case:
[mute@scelastic code]$ java test test.pem
PEM contents: [PEMKeyPair] [X509CertificateHolder] [X509CertificateHolder] [X509CertificateHolder]
cert0: CN=test.XXX.mil, OU=USN, OU=PKI, OU=DoD, O=U.S. Government, C=US
cert1: CN=DOD SW CA-60, OU=PKI, OU=DoD, O=U.S. Government, C=US
cert2: CN=DoD Root CA 3, OU=PKI, OU=DoD, O=U.S. Government, C=US
CA: CN=DoD Root CA 3, OU=PKI, OU=DoD, O=U.S. Government, C=US
java.security.cert.CertPathValidatorException: non-null policy tree required and policy tree is null
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:233)
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:141)
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
at test.main(test.java:52)
Caused by: java.security.cert.CertPathValidatorException: non-null policy tree required and policy tree is null
at sun.security.provider.certpath.PolicyChecker.processPolicies(PolicyChecker.java:572)
at sun.security.provider.certpath.PolicyChecker.checkPolicy(PolicyChecker.java:225)
at sun.security.provider.certpath.PolicyChecker.check(PolicyChecker.java:180)
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
... 5 more
org.bouncycastle.jce.exception.ExtCertPathValidatorException: No valid policy tree found when one expected.
at org.bouncycastle.jce.provider.RFC3280CertPathUtilities.processCertF(Unknown Source)
at org.bouncycastle.jce.provider.PKIXCertPathValidatorSpi.engineValidate(Unknown Source)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
at test.main(test.java:53)
java.lang.NullPointerException
at test.main(test.java:54)
I simply put in a chain.remove(ca); and
[mute@scelastic code]$ vi test.java
[mute@scelastic code]$ javac test.java && java test test.pem # go
PEM contents: [PEMKeyPair] [X509CertificateHolder] [X509CertificateHolder] [X509CertificateHolder]
cert0: CN=test.XXX.mil, OU=USN, OU=PKI, OU=DoD, O=U.S. Government, C=US
cert1: CN=DOD SW CA-60, OU=PKI, OU=DoD, O=U.S. Government, C=US
CA: CN=DoD Root CA 3, OU=PKI, OU=DoD, O=U.S. Government, C=US
JAVA: anyPolicy ROOT
2.16.840.1.101.2.1.11.36 CRIT: false EP: 2.16.840.1.101.2.1.11.36 (1)
2.16.840.1.101.2.1.11.36 CRIT: false EP: 2.16.840.1.101.2.1.11.36 (2)
BCP: 2.5.29.32.0 {
2.16.840.1.101.2.1.11.36 {
2.16.840.1.101.2.1.11.36 {
}
}
2.16.840.1.101.2.1.11.42 {
}
}
The relevant half of the test code:
public class test {
public static void main(String[] args) throws FileNotFoundException, CertificateException {
try {
List<X509Certificate> chain = createCertificateChain(args[0]);
X509Certificate ca = (X509Certificate)chain.get(chain.size() - 1);
PKIXCertPathValidatorResult r1 = null, r2 = null;
chain.remove(ca);
CertPath certPath = CertificateFactory.getInstance("X.509").generateCertPath(chain);
CertPathValidator v1 = CertPathValidator.getInstance("PKIX");
CertPathValidator v2 = CertPathValidator.getInstance("PKIX", new BouncyCastleProvider());
TrustAnchor trustAnchor = new TrustAnchor((X509Certificate)ca, null);
PKIXParameters params = new PKIXParameters(Collections.singleton(trustAnchor));
// params.setInitialPolicies(Collections.singleton("2.16.840.1.101.2.1.11.36"));
params.setRevocationEnabled(false);
for (int i = 0; i < chain.size(); i++) {
System.out.println("cert" + i + ": " + chain.get(i).getSubjectX500Principal());
}
System.out.println("CA: " + ca.getSubjectX500Principal());
try { r1 = (PKIXCertPathValidatorResult)v1.validate(certPath, params); } catch (Exception e) { e.printStackTrace(); }
try { r2 = (PKIXCertPathValidatorResult)v2.validate(certPath, params); } catch (Exception e) { e.printStackTrace(); }
System.out.println("JAVA: " + r1.getPolicyTree());
System.out.println("BCP: " + r2.getPolicyTree());
} catch (Exception e) {
e.printStackTrace();
}
}
But while I was hacking at the OpenJDK classes I commented out the Policy Checker and installed my cert into ECE and it's working OK.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.