The truststore does not contain any trusted certificate entries after upgrade to 8.0

[testshock]

I was doing offline upgrade from 7.16 to 8.0. With all servers stopped upgrade with rpm and started again.
From 7.16 to 7.17 no problems at all. Cluster came up green.
From 7.17 to 8.0 elastic i got

Exception
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - the truststore [/etc/elasticsearch/elastic-stack-ca.p12] does not contain any trusted certificate entries

The xpack part of my Elasticsearch.yml configuration:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: elastic-stack-ca.p12

Did i loose anything in Elasticsearch breaking changes | Elastic Installation and Upgrade Guide [8.0] | Elastic ?

[Tulsi (Tulsi Das)]

Did you have security feature enabled in your Older version?
String ES 8, security is enabled by default which can be disabled if not required.

[testshock]

Yes Tulsi
Security was enabled on 7.16-7.17 also.

[TimV (Tim Vernum)]

Hey, sorry this got missed from the breaking changes. We had a few improvements to the validation of SSL configuration, and this particular one got missed from the docs.

In your case, it looks like the old configuration worked by accident, because you're actually using the ca file as your keystore & truststore which is not what the docs advise, and it typically indicates an error in configuration.

In your case it looks like it worked (well enough) in 7.x, but in most case that sort of configuration would lead to additional, hard to diagnose errors. The change in 8.0 is designed to catch those errors earlier.

There's a couple of ways to solve this.

1. Extract the CA into a separate file and use that

keytool can extract the existing CA cert in PEM format for you, it would be something like:

keytool -exportcert -keystore elastic-stack-ca.p12 -storepass "" \
       -alias "ca" -rfc > elastic-stack-ca.pem

If your keystore has a password on it, then you will need to enter it in place of the quotes in -storepass "".

Then you can set

xpack.security.transport.ssl.certificate_authorities: elastic-stack-ca.pem

in your elasticsearch.yml

That will give you the same behaviour you had in ES7, although as I mentioned above, I think you've made a mistake in your configuration.

2. Stop using the CA file as your server cert

I would recommend that you fix up the problem with your nodes, rather than try and work around it.

It look like you're using a CA as the server certificate, which isn't ideal, and it's what the documentation guides you to do.

Typically you would use elasticsearch-certutil to create a CA (as you have done) and then use that CA to generate one or more server certificates for use in your nodes. If you follow those steps then the file you end up with is usable as a keystore and a truststore, while the CA file is not.

In these instructions: Set up basic security for the Elastic Stack | Elasticsearch Guide [8.0] | Elastic, under the Generate the certificate authority section, it looks like you did step 1 (elastic-stack-ca.p12) but not step 2 (elastic-certificates.p12)

So, then when you got to the Encrypt internode communications with TLS section, you used the
elastic-stack-ca.p12 file in place of the elastic-certificates.p12 file.

I would recommend going back through those instructions and working out where you went wrong, so you can switch over to using elastic-certificates.p12 instead.

[testshock]

Thank you Tim. I've followed your advice (2nd).
Green!