Trust two different certs on two nodes

Elastic Stack Elasticsearch
1

We are trying to add TLS security to our cluster. For compliance, all certs need to be generated via a in-house tool (they are signed by a 3rd party entity). I am currently testing with two nodes on my local box.

The cert I have is in a pfx/p12 format and it works when I have the same cert in the config folder for both nodes and have this in the config:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-cert.pfx
xpack.security.transport.ssl.truststore.path: elastic-cert.pfx

By compliance req the certs will get rotated on a specific cadence. The update will roll out one node at a time, so I need to handle having different certs on different nodes.

I am testing with two different certs in the two local nodes. Starting the first node works, but when I start the second node (which one starts first doesn't matter), I get this on the second node:

unable to find valid certification path to requested target

And this shows up on the first node:

client did not trust this server's certificate, closing connection

I have been looking through the docs and posts but my scenario seems different. I am not using ES to generate the certs, or creating the certs from a request generated by ES. I have my own cert.

How should I be going about enabling this?

Thanks

2

This doens't have to do with the fact that you generate your own certs versus using the elasticsearch utilities.

This has to do with the fact that:

  • You have a single keystore file to use both as a truststore ( containing the certificates that your node should trust ) and as a keystore ( containing the private key and the certificate that your node will use for TLS when connecting to or receiving connections from other nodes )
  • You will get this keystore updated for each node at a different time, but you seem to be using the same keystore for all nodes. This doesn't make much sense to me, unless I misunderstood what you were writing.

For compliance, all certs need to be generated via a in-house tool (they are signed by a 3rd party entity).

From the behavior you are describing above, it looks like the elastic-cert.pfx contains a private key and certificate entry, and the same certificate entry as a trusted certificate. When this keystore file is the same in all nodes as you start with, each node will be able to verify each other's certificate, as they all use the same one.

When you change the keystore in one only node (let's say node A):

  • If this node attempts to connect to node B, node B will present it's old certificate for TLS and node A can't trust that because you just configured it to only trust the new certificate.
  • If another node ( node B ) attempts to connect to node A, node A will present its new certificate for TLS but node A can't trust that as it knows nothing about it ( it is still configured to only trust the old certificate)

I will assume for now that you are(or will) actually get different pfx files for different nodes even to start with. Given your constraints, the only way I can think this work is that every time you get a new PFX for node N:

1.Export the trusted certificate from the elastic-cert.pfx, as a PEM certificate file, lets call it new_node_n.crt for now
2. Add the new_node_n.crt to the existing elasticsearch-cert.pfx on each one of the other nodes. Wait until all nodes have reloaded the file, we monitor the files and reload them if needed every 5 seconds by default.
3. On node N, overwrite the previous elasticsearch-cert.pfx with the new one. Wait until the node reloads the file ( ~5s )
4. Node N would only need to be restarted if the elasticsearch-cert.pfx is password protected and the password is stored in the elasticsearch keystore and the password needs to change.
5. Remove the old trusted keystore entry from elasticsearch-cert.pfx on all other nodes. In order to do that you need to keep track of the aliases in they keystore and know which is the new one and which is the old one.

Repeat steps above for every node, once you get a new certificate for them.

As you can see this gets a little challenging. May I offer two alternative approaches:

  • Use a specific subCA for Elasticsearch nodes TLS. This way you need only trust a specific CA certificate other than explicit certificates that rotate. You obviously need controls that ensure that this sub CA only issues certificates for elastisearch nodes.
  • Use self-signed certificates. I understand policies are policies, but there can be exceptions. Self signed certificates are totally fine for this use case, as the trust is explicit and there is no need to depend on your PKI and web of trust.

Finally:

I can guess that:

xpack.security.transport.ssl.verification_mode: certificate

is to facilitate your testing, but consider removing this in production so that the default value ( full ) takes effect. There is no reason for you not to do hostname verification.

3

This has helped a lot. There are a number of things I didn't understand here before working through your reply. Some of them I am still a bit unsure of. Also, I think I was wrong about being able to simply use the same cert on each node (as I can't get back to that state).

Say I have been given MyCert1.pfx and it's chain is:

RootCA
|->IntermediateCA
|->|-> MyCert1

If I have this on nodeA and I list:

xpack.security.transport.ssl.keystore.path: MyCert1.pfx
xpack.security.transport.ssl.truststore.path: MyCert1.pfx

When I start, I see this at /_xpack/ssl/certificates:

[
    {
        "path": "MyCert1.pfx",
        "format": "PKCS12",
        "alias": "37cced9a-63d6-41e4-993a-ee2ad82dee01",
        "subject_dn": "CN=foo.com, O=MyCorp, L=City, ST=State, C=US",
        "serial_number": "4e5c9247f2d880a5e511e1ecde8f9cd",
        "has_private_key": true,
        "expiry": "2021-09-18T12:00:00.000Z"
    }
]

If I try to start another node (nodeB) with the same cert, then I get:

nodeA: SSLHandshakeException: null cert chain
nodeB: Received fatal alert: bad_certificate

That makes sense I guess. I exported the RootCA and IntermediateCA and created a combined pfx that contained the full chain (fullChain.pfx). Now when I start nodeA I see this at /_xpack/ssl/certificates:

[
    {
        "path": "fullChain.pfx",
        "format": "PKCS12",
        "alias": "37cced9a-63d6-41e4-993a-ee2ad82dee01",
        "subject_dn": "CN=foo.com, O=MyCorp, L=City, ST=State, C=US",
        "serial_number": "4e5c9247f2d880a5e511e1ecde8f9ce",
        "has_private_key": true,
        "expiry": "2021-09-18T12:00:00.000Z"
    },
    {
        "path": "fullChain.pfx",
        "format": "PKCS12",
        "alias": "1",
        "subject_dn": "CN=My Root CA, OU=www.myCert.com, O=myCert Inc, C=US",
        "serial_number": "83be056904246b1a1756ac95991c74b",
        "has_private_key": false,
        "expiry": "2031-11-10T00:00:00.000Z"
    },
    {
        "path": "fullChain.pfx",
        "format": "PKCS12",
        "alias": "1",
        "subject_dn": "CN=myCert CA-1, O=myCert Inc, C=US",
        "serial_number": "19ec1c6bd3f597bb20c3338e551d878",
        "has_private_key": false,
        "expiry": "2030-08-04T12:00:00.000Z"
    }
]

When I start nodeB I still get the same errors. I would think that I am now presenting the full chain when making the connection, so I am confused as to why there is this error.

Ultimately, what I would like to do is to get to your recommendation:

Use a specific subCA for Elasticsearch nodes TLS. This way you need only trust a specific CA certificate other than explicit certificates that rotate. You obviously need controls that ensure that this sub CA only issues certificates for elastisearch nodes.

l am trying to figure out how to get there from here.

  • Does the root CA need to be in the truststore, or will the intermediate(s) suffice?

  • Does the cert that the node is going to present (i.e. the one in keystore) also need to be in the truststore, or is that only for validating certs from callers?

  • If I had the intermediate (and rootCA?) certs in the trust store, would that mean that it would trust any cert that came from that intermediate and I would not have to manage updating for each new cert as they are rolled out?

Thanks again for the detailed help!

4

To my last point, I think this is relevant:

The recommended approach for validating certificate authenticity in an Elasticsearch cluster is to trust the certificate authority (CA) that signed the certificate. By doing this, as nodes are added to your cluster they just need to use a certificate signed by the same CA and the node is automatically allowed to join the cluster.

I guess I just need to figure out how I go about trusting the CA that signed my certs.

5

And yet more that might relate.

Storing trusted certificates in a PKCS#12 file, although supported, is uncommon in practice. The elasticsearch-certutil tool, as well as Java’s keytool , are designed to generate PKCS#12 files that can be used both as a keystore and as a truststore, but this may not be the case for container files that are created using other tools.

6

and... because nothing is easy, I thought I would try this to use keytool to create a truststore:
https://docs.oracle.com/cd/E66686_01/pt855pbr1/eng/pt/tpst/task_ConfiguringSSLBetweenPeopleSoftAndElasticsearch.html?pli=ul_d73e131_tpst

When I attempt to import my CA step 6 from the above page (having skipped step 4), I get this when trying to import my pfx:

keytool error: java.security.cert.CertificateParsingException: signed fields invalid

And this when I convert it to a cer:

keytool error: java.security.cert.CertificateException: No certificate data found

7

Just checking back to see if anyone has any pointers on how I can trust my intermediate CA so that I can roll my certs out without breaking communications between nodes.

Thanks!

8

How about you export the CA certificate with

openssl pkcs12 -in your.pfx -cacerts -nokeys > ca.crt

if your pfx has both the root and the intermediate CA in there, chances are that you get both in the output, you need to keep only the intermediate one ( assuming this intermediate will be signing all your nodes certificates ).

Then in each one of your node, set

xpack.security.transport.ssl.keystore.path: MyCert1.pfx
xpack.security.transport.ssl.certificate_authorities: ["ca.crt"]
9

I'm still trying to get this to work. Right now I have the following trust store

-----BEGIN CERTIFICATE-----
intermediate cert
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
root cert
-----END CERTIFICATE-----

And keystore has:

-----BEGIN CERTIFICATE-----
my cert
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
intermediate cert
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
root cert
-----END CERTIFICATE-----

When I start the first node, if I go to:

_xpack/ssl/certificates

I see the full chain listed. When I start the second node (which is using the same keystore and truststore), I get:

io.netty.handler.codec.DecoderException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

I think this is related to this:
|||||||||||||||||||||||

The JSSE Reference Guide says this:

Storing trusted certificates in a PKCS12 keystore is not supported. PKCS12 is mainly used to deliver private keys with the associated certificate chains. It does not have any notion of "trusted" certificates. In terms of interoperability, other PKCS12 vendors have the same restriction. Browsers such as Mozilla and Internet Explorer do not accept a PKCS12 file with only trusted certificates.

Solution: Use the JKS keystore for storing trusted certificates.

This has changed a bit in Java 8, which supports trusted certificates in PKCS#12 - if they are marked with a special attribute (OID 2.16.840.1.113894.746875.1.1):

openssl pkcs12 -in microsoft.p12 -info
MAC Iteration 1024
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 1024
Certificate bag
Bag Attributes
    friendlyName: microsoft it ssl sha2 (baltimore cybertrust root)
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>

|||||||||||||||||||||||

When I look at the certs that are created with elasticsearch-certutil, I see the "Unsupported tag 6":

Certificate bag
Bag Attributes
    friendlyName: ca
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=CN = Elastic Certificate Tool Autogenerated CA

issuer=CN = Elastic Certificate Tool Autogenerated CA

-----BEGIN CERTIFICATE-----

I don't have that in my file (nor do I know how to get that). What I did was to import the cert into my windows cert store. I then opened it there and for each cert in the chain, I exported it. I then manually concatenated the certs into my "cert with key and full chain" and my "intermediate + root cert" certs.

So I guess the question is, if I have a cert, and in my Windows store, the full chain is trusted, how can I turn that into a p12 file that contains the chain as "trusted certificates?"

Thanks!

10

To add more context to the above, from the Security Settings in Elasticsearch doc already linked, there is this:

Storing trusted certificates in a PKCS#12 file, although supported, is uncommon in practice. The elasticsearch-certutil tool, as well as Java’s keytool , are designed to generate PKCS#12 files that can be used both as a keystore and as a truststore, but this may not be the case for container files that are created using other tools. Usually, PKCS#12 files only contain secret and private entries. To confirm that a PKCS#12 container includes trusted certificate ("anchor") entries look for 2.16.840.1.113894.746875.1.1: <Unsupported tag 6> in the openssl pkcs12 -info output, or trustedCertEntry in the keytool -list output.

11

I think I got it (at least I can use the same pfx from two nodes now without error). The key was in here:

First off, my cert did not have the full trust chain, so that was a problem. What resolved it for me was

Got a new sslCert for both Client and Server auth (Extended Key Usage)
I exported this to PEM as keyStore.pem:

openssl pkcs12 -in myCert.pfx -out keyStore.pem -nokeys -nodes

I edited the PEM file to remove out my cert and just keep the CAs.

Instead of specifying truststore, I added:

xpack.security.transport.ssl.certificate_authorities: [ "keyStore.pem" ]

When starting up and running _xpack/ssl/certificates, I see the CAs listed in both the PEM and pfx. Second node is able to join as well.

13

so I don't steer others off course, the correct command should have been:

openssl pkcs12 -in myCert.pfx -out keyStore.pem -nokeys -cacerts

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.