# Cloud UI / Proxy Certs **URL:** \ **Category:** Elastic Cloud Enterprise (ECE)\ **Created:** [June 13, 2020, 2:36am UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956 "2020-06-13T02:36:07Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![neutronscott](https://avatars.discourse-cdn.com/v4/letter/n/aca169/32.png) [@neutronscott](https://discuss.elastic.co/u/neutronscott)\ **Post date:** [June 13, 2020, 2:36am UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/1 "2020-06-13T02:36:07Z")
Has anyone actually gotten a cert to install? I always get "non-null policy tree required and policy tree is null" I have entered the docker container and replaced manually the 2 pems used by stunnel, but don't know where I'd manually place them to make them stick. ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [June 15, 2020, 2:12pm UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/2 "2020-06-15T14:12:53Z")
I assume that error you are getting is when uploading from the API or UI? This post has proven useful in the past in diagnosing why a cert has not been constructed "correctly" (where "correctly" means whatever Java library we use accepts it!): [ECE 2.0.1 Unable to upload Letsencrypt TLS certificate](https://discuss.elastic.co/t/ece-2-0-1-unable-to-upload-letsencrypt-tls-certificate/158802/3) Alex ---
**Author:** ![neutronscott](https://avatars.discourse-cdn.com/v4/letter/n/aca169/32.png) [@neutronscott](https://discuss.elastic.co/u/neutronscott)\ **Post date:** [June 15, 2020, 2:28pm UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/3 "2020-06-15T14:28:11Z")
Yes I have tried from UI and curl. It could be Java doesn't trust the root. It's an enterprise root. Is there a way to import Java cacerts into ECE? I also tried manually editing the Java cacerts but it may have not kept between service restarts (also not sure which container does this check) ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [June 15, 2020, 3:00pm UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/4 "2020-06-15T15:00:17Z")
Unfortunately there is currently no (persistent) way to edit the docker container's trusted CA list - to use an internal CA I believe it's necessary to terminate it at the load balancer that sits in front of the ECE proxies. (and then use the self-signed ECE certs, or external certs with a trusted CA, for the final hop). (I actually have no idea what would happen if you hand edited the `frc-admin-consoles-admin-console` services' `cacerts` for long enough to upload the certs, which is the only time we check their validity, it's not something we've ever tried) ---
**Author:** ![neutronscott](https://avatars.discourse-cdn.com/v4/letter/n/aca169/32.png) [@neutronscott](https://discuss.elastic.co/u/neutronscott)\ **Post date:** [June 16, 2020, 3:00am UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/5 "2020-06-16T03:00:46Z")
It's probably best I do setup a proxy now just to keep what I can behind port 443 for future ease with customer firewalls... but I'm stubborn and looked at this at home using Let's Encrypt's root **Fake LE Root X1** from the [staging environment](https://letsencrypt.org/docs/staging-environment/) which obviously isn't in the truststore. I entered the `frc-admin-consoles-admin-console` and added it like so: ```auto [root@scelastic ~]# docker exec -it frc-admin-consoles-admin-console bash [root@cec4e0674b75 /]# cd /etc/pki/ca-trust/source/anchors [root@cec4e0674b75 anchors]# cat >fakelerootx1.pem [root@cec4e0674b75 anchors]# update-ca-trust [root@cec4e0674b75 anchors]# pkill java ``` That worked. Survived a reboot anyway. It may cause trouble if other services need to trust the UI / Proxy cert, but the `ece ca master` wasn't in cacerts to begin with... So while **I** won't use this in production I may keep it in my lab to see what breaks. ---
**Author:** ![neutronscott](https://avatars.discourse-cdn.com/v4/letter/n/aca169/32.png) [@neutronscott](https://discuss.elastic.co/u/neutronscott)\ **Post date:** [June 17, 2020, 12:42am UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/6 "2020-06-17T00:42:46Z")
That was not the problem with our enterprise cert. ```auto [elastic@scelastic ~]$ curl -k -H Expect: -H 'Content-Type: application/json' --data-binary @./test.pem --user "admin:$pass" https://localhost:12443/api/v1/platform/configuration/security/tls/ui { "errors": [{ "code": "cert.invalid_cert_chain", "message": "Certificate chain was invalid [non-null policy tree required and policy tree is null]" }] } ``` ```auto [elastic@scelastic ~]$ curl -k -H Expect: -H 'Content-Type: application/json' --data-binary @./certbot/live/scelastic.domain/keychain.pem --user "admin:$pass" https://localhost:12443/api/v1/platform/configuration/security/tls/ui { } ``` But I have a proxy in front and get to also map the cluster UUIDs to meaningful names so at least there is that... ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [June 17, 2020, 1:46pm UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/7 "2020-06-17T13:46:12Z")
I had a look through our support database to see if anyone else had ever reported the specific `non-null policy tree` error and I found one hit, where (after lots of dead ends and red herrings): > [REDACTED] ultimately had to create their own certs because the intermediate they were using didn't comply with section 6.1.3 (section f) of RFC5280 which (fwiw!) [says](https://tools.ietf.org/html/rfc5280#section-6.1.3): ```auto (f) Verify that either explicit_policy is greater than 0 or the valid_policy_tree is not equal to NULL; ``` (For completeness, I also found one hit on the entire internet (!! well actually 2 but the other one was you 🙂 ): [SSLHandshakeException - java.security.cert.CertPathValidatorException: non-null policy tree required and policy tree is null - Stack Overflow](https://stackoverflow.com/questions/21474847/sslhandshakeexception-java-security-cert-certpathvalidatorexception-non-null), but alas the post never got answered) ---
**Author:** ![neutronscott](https://avatars.discourse-cdn.com/v4/letter/n/aca169/32.png) [@neutronscott](https://discuss.elastic.co/u/neutronscott)\ **Post date:** [June 17, 2020, 3:02pm UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/8 "2020-06-17T15:02:04Z")
Thank you for continuing to look at this. Since I'm the only one in the world with the problem I'll continue documenting what I found to avoid a [xkcd: Wisdom of the Ancients](https://xkcd.com/979/) situation. 🙂 I set `-Djava.security.debug=all` in my `/service/adminconsole/run` java command line. Here is final page of side-by-side diff of my Fake LE cert on the left and a DOD cert on the right. ![Screen Shot 2020-06-17 at 10.34.10 AM](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40a8d7606ecba77d051406972427c56130b93a76.png) The application doesn't care about a policy so validator should be checking for anyPolicy. But they aren't ending up in valid\_policy\_three and throws an exception in PolicyChecker.processPolicies() no matter if initPolicies was anyPolicy: ```auto 511 // PKIX: Section 6.1.3: Step (f) 512 // verify that either explicit policy is greater than 0 or 513 // the valid_policy_tree is not equal to NULL 514 if ((explicitPolicy == 0) && (rootNode == null)) { 515 throw new CertPathValidatorException 516 ("non-null policy tree required and policy tree is null", 517 null, null, -1, PKIXReason.INVALID_POLICY); 518 } ``` A similar issue with certpath verification was found in NSS by Chrome: [Issue 545103 - Work around the SEC\_ERROR\_POLICY\_VALIDATION\_FAILED error from CERT\_PKIXVerifyCert by retrying CERT\_PKIXVerifyCert with the certificate policy in the certificate.](https://codereview.chromium.org/545103/) Reported upstream: [552775 - CERT\_PKIXVerifyCert fails with SEC\_ERROR\_POLICY\_VALIDATION\_FAILED if an intermediate CA cert has requireExplicitPolicy](https://bugzilla.mozilla.org/show_bug.cgi?id=552775) I feel like it should not do that if initialPolicies was empty. The Chrome workaround was to go back through and check it for the policy on the leaf certificate. I say this is a Java bug in `sun.security.provider.certpath.PolicyChecker` ---
**Author:** ![neutronscott](https://avatars.discourse-cdn.com/v4/letter/n/aca169/32.png) [@neutronscott](https://discuss.elastic.co/u/neutronscott)\ **Post date:** [June 17, 2020, 4:43pm UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/9 "2020-06-17T16:43:29Z")
After reviewing how SSLSockets validate certs, I think ECE could use a simplier validator and is likely the best solution to this problem. I probably will have to wait until our support contract starts to open the bug report? Very new to elastic (and java).. Looks like eventually no.found.security.certificate.CertChain.apply calls the `PKIX` validator when `Validator.VAR_TLS_SERVER` is probably more appropriate. ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [June 17, 2020, 6:59pm UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/10 "2020-06-17T18:59:18Z")
I opened an issue in our developer repo to investigate alternative validators, for our API team, but if/when you do have a support contract, please do raise an issue since we use those to help with our prioritization Alex ---
**Author:** ![neutronscott](https://avatars.discourse-cdn.com/v4/letter/n/aca169/32.png) [@neutronscott](https://discuss.elastic.co/u/neutronscott)\ **Post date:** [June 19, 2020, 7:10am UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/11 "2020-06-19T07:10:15Z")
After future digging and hacking away at the OpenJDK classes ... I found this is not a bug in Java. Further inspection of RFC shows we should start path checks at the intermediate and things in the trust anchor needn't be checked. > When the trust anchor is provided in the form of a self-signed certificate, this self-signed certificate is not included as part of the prospective certification path. Information about trust anchors is provided as inputs to the certification path validation algorithm ([Section 6.1.1](https://tools.ietf.org/html/rfc5280#section-6.1.1)). So all that needs done is to remove the ca from the chain before calling `generateCertPath()` I crafted a test case: ```auto [mute@scelastic code]$ java test test.pem PEM contents: [PEMKeyPair] [X509CertificateHolder] [X509CertificateHolder] [X509CertificateHolder] cert0: CN=test.XXX.mil, OU=USN, OU=PKI, OU=DoD, O=U.S. Government, C=US cert1: CN=DOD SW CA-60, OU=PKI, OU=DoD, O=U.S. Government, C=US cert2: CN=DoD Root CA 3, OU=PKI, OU=DoD, O=U.S. Government, C=US CA: CN=DoD Root CA 3, OU=PKI, OU=DoD, O=U.S. Government, C=US java.security.cert.CertPathValidatorException: non-null policy tree required and policy tree is null at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135) at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:233) at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:141) at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80) at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) at test.main(test.java:52) Caused by: java.security.cert.CertPathValidatorException: non-null policy tree required and policy tree is null at sun.security.provider.certpath.PolicyChecker.processPolicies(PolicyChecker.java:572) at sun.security.provider.certpath.PolicyChecker.checkPolicy(PolicyChecker.java:225) at sun.security.provider.certpath.PolicyChecker.check(PolicyChecker.java:180) at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ... 5 more org.bouncycastle.jce.exception.ExtCertPathValidatorException: No valid policy tree found when one expected. at org.bouncycastle.jce.provider.RFC3280CertPathUtilities.processCertF(Unknown Source) at org.bouncycastle.jce.provider.PKIXCertPathValidatorSpi.engineValidate(Unknown Source) at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292) at test.main(test.java:53) java.lang.NullPointerException at test.main(test.java:54) ``` I simply put in a `chain.remove(ca);` and ```auto [mute@scelastic code]$ vi test.java [mute@scelastic code]$ javac test.java && java test test.pem # go PEM contents: [PEMKeyPair] [X509CertificateHolder] [X509CertificateHolder] [X509CertificateHolder] cert0: CN=test.XXX.mil, OU=USN, OU=PKI, OU=DoD, O=U.S. Government, C=US cert1: CN=DOD SW CA-60, OU=PKI, OU=DoD, O=U.S. Government, C=US CA: CN=DoD Root CA 3, OU=PKI, OU=DoD, O=U.S. Government, C=US JAVA: anyPolicy ROOT 2.16.840.1.101.2.1.11.36 CRIT: false EP: 2.16.840.1.101.2.1.11.36 (1) 2.16.840.1.101.2.1.11.36 CRIT: false EP: 2.16.840.1.101.2.1.11.36 (2) BCP: 2.5.29.32.0 { 2.16.840.1.101.2.1.11.36 { 2.16.840.1.101.2.1.11.36 { } } 2.16.840.1.101.2.1.11.42 { } } ``` The relevant half of the test code: ```auto public class test { public static void main(String[] args) throws FileNotFoundException, CertificateException { try { List chain = createCertificateChain(args[0]); X509Certificate ca = (X509Certificate)chain.get(chain.size() - 1); PKIXCertPathValidatorResult r1 = null, r2 = null; chain.remove(ca); CertPath certPath = CertificateFactory.getInstance("X.509").generateCertPath(chain); CertPathValidator v1 = CertPathValidator.getInstance("PKIX"); CertPathValidator v2 = CertPathValidator.getInstance("PKIX", new BouncyCastleProvider()); TrustAnchor trustAnchor = new TrustAnchor((X509Certificate)ca, null); PKIXParameters params = new PKIXParameters(Collections.singleton(trustAnchor)); // params.setInitialPolicies(Collections.singleton("2.16.840.1.101.2.1.11.36")); params.setRevocationEnabled(false); for (int i = 0; i < chain.size(); i++) { System.out.println("cert" + i + ": " + chain.get(i).getSubjectX500Principal()); } System.out.println("CA: " + ca.getSubjectX500Principal()); try { r1 = (PKIXCertPathValidatorResult)v1.validate(certPath, params); } catch (Exception e) { e.printStackTrace(); } try { r2 = (PKIXCertPathValidatorResult)v2.validate(certPath, params); } catch (Exception e) { e.printStackTrace(); } System.out.println("JAVA: " + r1.getPolicyTree()); System.out.println("BCP: " + r2.getPolicyTree()); } catch (Exception e) { e.printStackTrace(); } } ``` But while I was hacking at the OpenJDK classes I commented out the Policy Checker and installed my cert into ECE and it's working OK. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 3, 2020, 7:22am UTC](https://discuss.elastic.co/t/cloud-ui-proxy-certs/236956/12 "2020-07-03T07:22:17Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.