I set up a test instance of elasticsearch 7.11.1 on a Windows VM and created the certificates for SSL encryption following the processes in the documentation, which worked without an issue.
I then created CSR's for all nodes and sent that to my local CA provider to create certificates. Using certificates provided by the local CA the nodes will not connect.
I get the following error:
the server provided a certificate with subject name [<CERTIFICATE INFO>]
and fingerprint [XXXXXXXXXXXX]; the certificate has subject alternative names [<DNS AND IP>];
the certificate is issued by [<ISSUING CA>]
but the server did not provide a copy of the issuing certificate in the certificate chain;
this ssl context ([xpack.security.transport.ssl]) is not configured to trust that issuer,
it only trusts the issuer [<DIFFERENT THAN ISSUING CA ABOVE>]
with fingerprint [XXXXXXXXXXXX]
SSL CONFIGURATION in yaml file:
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: server.p12
xpack.security.transport.ssl.truststore.path: server.p12
xpack.security.transport.ssl.verification_mode: full
The certificates are serverAuth and clientAuth
Private keys generated in the CSR request were not used when certificates were created.
They were provided to me in pkcs12 format
Certs include DNS and IP.
Cert passwords stored in elasticsearch.keystore
Root CA files were provided in .pem format, and I added them to the Trusted root providers on the servers. How do I get a copy of the issuing certificate in the certificate chain? Is this a configuration issue or a root cert location issue?
Explicitly trust the issuer cert by adding it to the truststore. You can check whether it is there with openssl pkcs12 -info -in server.p12 -nokeys
Or having it send through during TLS handshake (assuming it is issued by your root CA).
Btw,
It is recommended to use separate CA and certs for inter-node communication. That is, it is perfect valid to keep using the CA and you generated by following the documentation. Using a shared CA is in fact not recommended because trusting a shared CA may allow other people to spin up a node and join your cluster if he/she has a cert signed by the shared CA.
v7.11 is no longer supported. I suggested you upgrade to latest 8.x as soon as possible which features automatically setting up security.
I checked the cert and although they said it had the cert for the CA it did not. I had a .pem file for the CA so I used that to add the CA to the p12 cert which I am using as the truststore, however that did not work. Still working on learning all the ins and outs of SSL and certificates, if you have any other suggestions I am open to them.
Appreciate all your advice. My company does not want me using self-signed certificates or certs from another CA. So, I am currently stuck with the using company certs and waiting to upgrade. The certificates have both CAs in the chain.
This is where I am now:
DiagnosticTrustManager: failed to establish trust with server at [master node];
server provided a certificate with subject name [master cert info (three DC's)] and fingerprint [xxxx] ;
the certificate has subject alternative names [DNS full, DNS compname, IP];
the certificate is issued by [company CA (two DC's)];
the certificate is signed by (subject [Main Company Root CA (two DC's)] fingerprint [xxxxx]) which is self-issued:
the [main company Root CA] certificate is not trusted in this context.
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
Not sure it matters but the last two DC's in all the certs match.
I have been reviewing posts with this issue, but haven't found a post that seems to match my issue for connections between nodes.
You should check whether the required certificate is indeed available in the server.p12 file. You can compare the fingerprints to the one you got in the error message. Something like the follows:
Use openssl to export certs from server.p12, e.g. openssl pkcs12 -info -in server.p12 -nokeys > all.certs
Split the certs from above all.certs into their own files (each file should start with -----BEGIN CERTIFICATE----- and end with -----END CERTIFICATE-----
For each of the cert file, check their fingerprint with openssl x509 -fingerprint -noout -in YOUR_CERT_FILE.
My guess is that none of the fingerprint would match the one in the error message. If that's the case, you need talk to your local CA to grab the correct certificate and add it to the truststore.
I have checked the fingerprints of all included certificates and they match the three listed:
The server cert, the root cert, and the cert issued from the root cert.
server.p12 has all three included with matching fingerprints.
It is definitely a xpack.security.transport.ssl issue (according to the logs).
I do not think the private key for each root certificate is included, but I am checking.
Do the included root certificates need the private key?
Can Elasticsearch xpack handle a certificate issued from a secondary root certificate authority?
It would be helpful if you can give exact information. It's hard to just talk about server cert, root cert, ca cert and cross compare them especially since the mismatch is not in the names, but the fingerprints. Certificate fingerprints are not secrets. The entire certificate is meant to be public. Could you please share the original error that include the excat fingerprints and your fingerprint checks of the server.p12 file so that we can be sure that we are not cross talking?
Do the included root certificates need the private key?
No. You don't need CA key for this to work. It is actually recommended to keep your CA key separate.
Can Elasticsearch xpack handle a certificate issued from a secondary root certificate authority?
Thank you again for continuing the conversation and your patience.
The error is the same as the above:
DiagnosticTrustManager: failed to establish trust with server at [master node];
server provided a certificate with subject name [master cert info] and fingerprint [xxxx] ;
the certificate has subject alternative names [DNS full, DNS compname, IP];
the certificate is issued by [company CA (two DC's)];
the certificate is signed by (subject [Main Company Root CA (two DC's)] fingerprint [xxxxx]) which is self-issued:
the [main company Root CA] certificate is not trusted in this context.
related to xpack.security.transport.ssl
Fingerprints of each in the order they appear in the error message:
server cert: FE:63:31:DB:89:34:26:18:D3:3D:05:8D:A3:09:1D:FA:9B:F1:7F:F2
from error: fe6331db89342618d33d058da3091dfa9bf17ff2
cert issued by: 18:9C:0E:90:53:10:26:44:21:81:16:88:EC:CC:5E:51:3D:0F:3C:91
from error: 189c0e905310264421811688eccc5e513d0f3c91
cert signed by: 84:6D:D1:25:59:E7:EC:1F:40:51:71:8E:32:4B:CE:7C:1E:31:2F:83
from error: 846dd12559e7ec1f4051718e324bce7c1e312f83
I don't think I can post exact IP's and other information from the error code. Let me know if there is not enough information.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.