Cannot add new CA to keystore

[alrubaa (Ahmad)]

Hi All,

I have an ELasticsearch cluster of 12 nodes running 8.2 and the certificates have expired, just crossed 3 years which I did not realise. I have been trying to follow the instructions on Update security certificates with a different CA | Elasticsearch Guide [8.6] | Elastic but I hit a glitch.
I can generate the new CA.

My .yml has:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

When I come to add the new CA to the keystore using the command:
sudo keytool -importcert -trustcacerts -noprompt -keystore /etc/elasticsearch/elastic-certificates.p12 -alias 2023-ca -file ca.crt

I get asked for a password that I do not seem to get right. When I created elastic-certificates.p12 certificate years ago I did not use a password, and other passwords I generated do not seem to work.

I tried: bin/elasticsearch-keystore passwd, but it did not help.

I tried: bin/elasticsearch-keystore show xpack.security.transport.ssl.keystore.password

But I get ERROR: Setting [xpack.security.transport.ssl.keystore.password] does not exist in the keystore.

However, they keytool command does not allow me to proceed without a 6 character password.

I am stuck, help please, how can I add the new CA to my keystore and what am I missing?
Thank you.

[Yang_Wang]

If you have the transport SSL password stored in the ES keystore file (elasticsearch.keystore), its name should be xpack.security.transport.ssl.keystore.secure_password. So you'll need

./bin/elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password

You can also check all the keys in the ES keystore file with ./bin/elasticsearch-keystore list.

[alrubaa (Ahmad)]

Thank you for that, I already tried all of that. The issue is related to the step in the Elasticsearch instructions I referred to above that states use keytool to trust the new CA. keytool must have a password and does not accept empty passwords and I do not have a password for the keystore. So I could not proceed. Adding a password to the keystore still did not solve the issue. The question now is whether the step to get the cluster to trust the new CA is actually required if all certificates are being replaced because the CA has expired?
I tried on a dev cluster, where I generated new CA and new certificates and merely replaced all certificates on all nodes, everything seems to work. Does that make sense, in which case why does the documentation has an extra step?

[Yang_Wang]

@alrubaa

Adding a password to the keystore still did not solve the issue.

Why that was the case? What error did you encounter? You can add a password to a keystore file with empty password with a command like

keytool -storepasswd -keystore YOUR_KEYSTORE.p12 -new YOUR_NEW_PASSWORD -storepass ''

why does the documentation has an extra step

The instruction given in the documentation is to avoid complete downtime of the cluster when replacing close-to-expire CA certs. This is what people normally wants in a production environment. But if you could afford some downtime or if the CA has already expired, you can consider to skip this step.