# Trust two different certs on two nodes **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-security\ **Created:** [June 3, 2020, 9:57pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642 "2020-06-03T21:57:47Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 3, 2020, 9:57pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/1 "2020-06-03T21:57:47Z")
We are trying to add TLS security to our cluster. For compliance, all certs need to be generated via a in-house tool (they are signed by a 3rd party entity). I am currently testing with two nodes on my local box. The cert I have is in a pfx/p12 format and it works when I have the same cert in the config folder for both nodes and have this in the config: ``` xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: elastic-cert.pfx xpack.security.transport.ssl.truststore.path: elastic-cert.pfx ``` By compliance req the certs will get rotated on a specific cadence. The update will roll out one node at a time, so I need to handle having different certs on different nodes. I am testing with two different certs in the two local nodes. Starting the first node works, but when I start the second node (which one starts first doesn't matter), I get this on the second node: `unable to find valid certification path to requested target` And this shows up on the first node: `client did not trust this server's certificate, closing connection` I have been looking through the docs and posts but my scenario seems different. I am not using ES to generate the certs, or creating the certs from a request generated by ES. I have my own cert. How should I be going about enabling this? Thanks ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [June 4, 2020, 5:45am UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/2 "2020-06-04T05:45:57Z")
> [@jthoni](#): > > I have been looking through the docs and posts but my scenario seems different. I am not using ES to generate the certs, or creating the certs from a request generated by ES. I have my own cert. This doens't have to do with the fact that you generate your own certs versus using the elasticsearch utilities. This has to do with the fact that: - You have a single keystore file to use both as a truststore ( containing the certificates that your node should trust ) and as a keystore ( containing the private key and the certificate that your node will use for TLS when connecting to or receiving connections from other nodes ) - You will get this keystore updated for each node at a different time, but you seem to be using the same keystore for all nodes. This doesn't make much sense to me, unless I misunderstood what you were writing. > For compliance, all certs need to be generated via a in-house tool (they are signed by a 3rd party entity). From the behavior you are describing above, it looks like the `elastic-cert.pfx` contains a private key and certificate entry, and the same certificate entry as a trusted certificate. When this keystore file is the same in all nodes as you start with, each node will be able to verify each other's certificate, as they _all_ use the _same_ one. When you change the keystore in _one_ only node (let's say node A): - If this node attempts to connect to node B, node B will present it's old certificate for TLS and node A can't trust that because you just configured it to _only_ trust the new certificate. - If another node ( node B ) attempts to connect to node A, node A will present its new certificate for TLS but node A can't trust that as it knows nothing about it ( it is still configured to only trust the old certificate) I will assume for now that you are(or will) actually get different pfx files for different nodes even to start with. Given your constraints, the only way I can think this work is that every time you get a new PFX for node N: 1.Export the trusted certificate from the elastic-cert.pfx, as a PEM certificate file, lets call it `new_node_n.crt` for now 2. Add the `new_node_n.crt` to the _existing_ `elasticsearch-cert.pfx` on each one of the other nodes. Wait until all nodes have reloaded the file, we monitor the files and reload them if needed every 5 seconds by default. 3. On node N, overwrite the previous `elasticsearch-cert.pfx` with the new one. Wait until the node reloads the file ( ~5s ) 4. Node N would only need to be restarted if the `elasticsearch-cert.pfx` is password protected **and** the password is stored in the elasticsearch keystore **and** the password needs to change. 5. Remove the old trusted keystore entry from `elasticsearch-cert.pfx` on all other nodes. In order to do that you need to keep track of the aliases in they keystore and know which is the new one and which is the old one. Repeat steps above for every node, once you get a new certificate for them. As you can see this gets a little challenging. May I offer two alternative approaches: - Use a specific subCA for Elasticsearch nodes TLS. This way you need only trust a specific CA certificate other than explicit certificates that rotate. You obviously need controls that ensure that this sub CA _only_ issues certificates for elastisearch nodes. - Use self-signed certificates. I understand policies are policies, but there can be exceptions. Self signed certificates are totally fine for this use case, as the trust is explicit and there is no need to depend on your PKI and web of trust. Finally: I can guess that: ```auto xpack.security.transport.ssl.verification_mode: certificate ``` is to facilitate your testing, but consider removing this in production so that the default value ( `full` ) takes effect. There is no reason for you not to do hostname verification. ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 5, 2020, 11:26pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/3 "2020-06-05T23:26:24Z")
This has helped a lot. There are a number of things I didn't understand here before working through your reply. Some of them I am still a bit unsure of. Also, I think I was wrong about being able to simply use the same cert on each node (as I can't get back to that state). Say I have been given MyCert1.pfx and it's chain is: RootCA |-\>IntermediateCA |-\>|-\> MyCert1 If I have this on nodeA and I list: ``` xpack.security.transport.ssl.keystore.path: MyCert1.pfx xpack.security.transport.ssl.truststore.path: MyCert1.pfx ``` When I start, I see this at /\_xpack/ssl/certificates: ``` [ { "path": "MyCert1.pfx", "format": "PKCS12", "alias": "37cced9a-63d6-41e4-993a-ee2ad82dee01", "subject_dn": "CN=foo.com, O=MyCorp, L=City, ST=State, C=US", "serial_number": "4e5c9247f2d880a5e511e1ecde8f9cd", "has_private_key": true, "expiry": "2021-09-18T12:00:00.000Z" } ] ``` If I try to start another node (nodeB) with the same cert, then I get: nodeA: SSLHandshakeException: null cert chain nodeB: Received fatal alert: bad\_certificate That makes sense I guess. I exported the RootCA and IntermediateCA and created a combined pfx that contained the full chain (fullChain.pfx). Now when I start nodeA I see this at /\_xpack/ssl/certificates: ``` [ { "path": "fullChain.pfx", "format": "PKCS12", "alias": "37cced9a-63d6-41e4-993a-ee2ad82dee01", "subject_dn": "CN=foo.com, O=MyCorp, L=City, ST=State, C=US", "serial_number": "4e5c9247f2d880a5e511e1ecde8f9ce", "has_private_key": true, "expiry": "2021-09-18T12:00:00.000Z" }, { "path": "fullChain.pfx", "format": "PKCS12", "alias": "1", "subject_dn": "CN=My Root CA, OU=www.myCert.com, O=myCert Inc, C=US", "serial_number": "83be056904246b1a1756ac95991c74b", "has_private_key": false, "expiry": "2031-11-10T00:00:00.000Z" }, { "path": "fullChain.pfx", "format": "PKCS12", "alias": "1", "subject_dn": "CN=myCert CA-1, O=myCert Inc, C=US", "serial_number": "19ec1c6bd3f597bb20c3338e551d878", "has_private_key": false, "expiry": "2030-08-04T12:00:00.000Z" } ] ``` When I start nodeB I still get the same errors. I would think that I am now presenting the full chain when making the connection, so I am confused as to why there is this error. Ultimately, what I would like to do is to get to your recommendation: > Use a specific subCA for Elasticsearch nodes TLS. This way you need only trust a specific CA certificate other than explicit certificates that rotate. You obviously need controls that ensure that this sub CA _only_ issues certificates for elastisearch nodes. l am trying to figure out how to get there from here. - Does the root CA need to be in the truststore, or will the intermediate(s) suffice? - Does the cert that the node is going to present (i.e. the one in keystore) also need to be in the truststore, or is that only for validating certs from callers? - If I had the intermediate (and rootCA?) certs in the trust store, would that mean that it would trust any cert that came from that intermediate and I would not have to manage updating for each new cert as they are rolled out? Thanks again for the detailed help! ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 5, 2020, 11:39pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/4 "2020-06-05T23:39:17Z")
To my last point, I think this is relevant: > **[Configure TLS | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html)** > The recommended approach for validating certificate authenticity in an Elasticsearch cluster is to trust the certificate authority (CA) that signed the certificate. By doing this, as nodes are added to your cluster they just need to use a certificate signed by the same CA and the node is automatically allowed to join the cluster. I guess I just need to figure out how I go about trusting the CA that signed my certs. ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 6, 2020, 12:04am UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/5 "2020-06-06T00:04:32Z")
And yet more that might relate. > **[Security settings in Elasticsearch | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html)** > Storing trusted certificates in a PKCS#12 file, although supported, is uncommon in practice. The [`elasticsearch-certutil` tool](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html), as well as Java’s `keytool` , are designed to generate PKCS#12 files that can be used both as a keystore and as a truststore, but this may not be the case for container files that are created using other tools. ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 6, 2020, 12:33am UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/6 "2020-06-06T00:33:28Z")
and... because nothing is easy, I thought I would try this to use keytool to create a truststore: [https://docs.oracle.com/cd/E66686\_01/pt855pbr1/eng/pt/tpst/task\_ConfiguringSSLBetweenPeopleSoftAndElasticsearch.html?pli=ul\_d73e131\_tpst](https://docs.oracle.com/cd/E66686_01/pt855pbr1/eng/pt/tpst/task_ConfiguringSSLBetweenPeopleSoftAndElasticsearch.html?pli=ul_d73e131_tpst) When I attempt to import my CA step 6 from the above page (having skipped step 4), I get this when trying to import my pfx: `keytool error: java.security.cert.CertificateParsingException: signed fields invalid` And this when I convert it to a cer: `keytool error: java.security.cert.CertificateException: No certificate data found` ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 8, 2020, 6:44pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/7 "2020-06-08T18:44:13Z")
Just checking back to see if anyone has any pointers on how I can trust my intermediate CA so that I can roll my certs out without breaking communications between nodes. Thanks! ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [June 9, 2020, 12:20pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/8 "2020-06-09T12:20:42Z")
How about you export the CA certificate with ```auto openssl pkcs12 -in your.pfx -cacerts -nokeys > ca.crt ``` if your pfx has both the root and the intermediate CA in there, chances are that you get both in the output, you need to keep only the intermediate one ( assuming this intermediate will be signing all your nodes certificates ). Then in each one of your node, set ```auto xpack.security.transport.ssl.keystore.path: MyCert1.pfx xpack.security.transport.ssl.certificate_authorities: ["ca.crt"] ``` ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 16, 2020, 3:34am UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/9 "2020-06-16T03:34:27Z")
I'm still trying to get this to work. Right now I have the following trust store ``` -----BEGIN CERTIFICATE----- intermediate cert -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- root cert -----END CERTIFICATE----- ``` And keystore has: ``` -----BEGIN CERTIFICATE----- my cert -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- intermediate cert -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- root cert -----END CERTIFICATE----- ``` When I start the first node, if I go to: \_`xpack/ssl/certificates` I see the full chain listed. When I start the second node (which is using the same keystore and truststore), I get: `io.netty.handler.codec.DecoderException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` I think this is related to this: ||||||||||||||||||||||| > > > Hello, > > I have a problem using a PKCS #12 created with Portecle, which is using …a previous version of Bouncy Castle (1.50 and 1.52) and \[portecle\](http://portecle.sourceforge.net/). > Here is a PKCS #12 keystore containing Goolge’s www.google.fr:443 TLS certificate \[google.p12.zip\](https://github.com/kaikramer/keystore-explorer/files/151098/google.p12.zip) (password: 123456). > It is visible using GNOME default tool: GCR-viewer. But is not seen by keystore-explorer. > I watched the alias returned by the load method, but no alias exists. > > \`\`\` java > keyStore.load(fis, password.toCharArray()); > Enumeration\ a = keyStore.aliases(); > while (a.hasMoreElements()) { > String aliasz = (String) a.nextElement(); > System.out.println("Alias : " + aliasz); > } > \`\`\` > > I have not found a way to know if the loaded PKCS #12 can hide unrecognized keys. > I’d like to inform the user with an information message to indicate that the PKCS #12 is kinda corrupted. > > Maybe you have an idea for it. > > Thanks, > > PS : When using version 1.52 of Bouncy castle with portecle, it does not happen to read the PKCS #12. > > Luc The [JSSE Reference Guide](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html) says this: > Storing trusted certificates in a PKCS12 keystore is not supported. PKCS12 is mainly used to deliver private keys with the associated certificate chains. It does not have any notion of "trusted" certificates. In terms of interoperability, other PKCS12 vendors have the same restriction. Browsers such as Mozilla and Internet Explorer do not accept a PKCS12 file with only trusted certificates. > > Solution: Use the JKS keystore for storing trusted certificates. This has changed a bit in Java 8, which supports trusted certificates in PKCS#12 - if they are marked with a special attribute (OID 2.16.840.1.113894.746875.1.1): ```auto openssl pkcs12 -in microsoft.p12 -info MAC Iteration 1024 MAC verified OK PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 1024 Certificate bag Bag Attributes friendlyName: microsoft it ssl sha2 (baltimore cybertrust root) 2.16.840.1.113894.746875.1.1: ``` ||||||||||||||||||||||| When I look at the certs that are created with elasticsearch-certutil, I see the "Unsupported tag 6": ``` Certificate bag Bag Attributes friendlyName: ca 2.16.840.1.113894.746875.1.1: subject=CN = Elastic Certificate Tool Autogenerated CA issuer=CN = Elastic Certificate Tool Autogenerated CA -----BEGIN CERTIFICATE----- ``` I don't have that in my file (nor do I know how to get that). What I did was to import the cert into my windows cert store. I then opened it there and for each cert in the chain, I exported it. I then manually concatenated the certs into my "cert with key and full chain" and my "intermediate + root cert" certs. So I guess the question is, if I have a cert, and in my Windows store, the full chain is trusted, how can I turn that into a p12 file that contains the chain as "trusted certificates?" Thanks! ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 16, 2020, 3:55pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/10 "2020-06-16T15:55:38Z")
To add more context to the above, from the Security Settings in Elasticsearch doc already linked, there is this: > Storing trusted certificates in a PKCS#12 file, although supported, is uncommon in practice. The [`elasticsearch-certutil` tool](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html), as well as Java’s `keytool` , are designed to generate PKCS#12 files that can be used both as a keystore and as a truststore, but this may not be the case for container files that are created using other tools. Usually, PKCS#12 files only contain secret and private entries. To confirm that a PKCS#12 container includes trusted certificate ("anchor") entries look for `2.16.840.1.113894.746875.1.1: ` in the `openssl pkcs12 -info` output, or `trustedCertEntry` in the `keytool -list` output. ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 16, 2020, 8:17pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/11 "2020-06-16T20:17:34Z")
I think I got it (at least I can use the same pfx from two nodes now without error). The key was in here: > [@SSLHandshakeException when enabling security](https://discuss.elastic.co/t/sslhandshakeexception-when-enabling-security/186680): > > Hi we have a working cluster but when we enable xpack security the transport communication fails. Configuration: bootstrap.memory\_lock: true cluster.initial\_master\_nodes: - elk.mydomain.dk - elk-02.mydomain.dk - elk-03.mydomain.dk cluster.name: elasticsearch-prod discovery.seed\_hosts: - elk-02.mydomain.dk - elk-03.mydomain.dk http.port: 9200 network.host: elk.mydomain.dk node.data: false node.ingest: true node.master: true node.max\_local\_storage\_nodes: 1 node.name: ELK path.dat… First off, my cert did not have the full trust chain, so that was a problem. What resolved it for me was Got a new sslCert for both Client and Server auth (Extended Key Usage) I exported this to PEM as keyStore.pem: `openssl pkcs12 -in myCert.pfx -out keyStore.pem -nokeys -nodes` I edited the PEM file to remove out my cert and just keep the CAs. Instead of specifying truststore, I added: xpack.security.transport.ssl.certificate\_authorities: ["keyStore.pem"] When starting up and running \_xpack/ssl/certificates, I see the CAs listed in both the PEM and pfx. Second node is able to join as well. ---
**Author:** ![jthoni](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jthoni](https://discuss.elastic.co/u/jthoni)\ **Post date:** [June 17, 2020, 6:04pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/13 "2020-06-17T18:04:55Z")
so I don't steer others off course, the correct command should have been: openssl pkcs12 -in myCert.pfx -out keyStore.pem -nokeys -cacerts ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 15, 2020, 6:08pm UTC](https://discuss.elastic.co/t/trust-two-different-certs-on-two-nodes/235642/14 "2020-07-15T18:08:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.