Updating xpack certificates to a new CA in an active Elasticsearch cluster without downtime

Elastic Stack Elasticsearch
16

@DavidTurner - We've completed thorough validation and identified the root cause.

Important Context:

We are NOT attempting certificate migration, all nodes currently use certificates from the same issuer. We're simply testing whether we can move from keystore-only to truststore with the current certificates.

Test Configuration:

  • MASTER node: keystore-only - current production configuration

  • QUERY node: keystore + truststore - testing trust with same root CAs

Truststore Verification:

We verified the truststore contains the exact same root CAs as the keystore by comparing SHA256 fingerprints between both files - they match perfectly (verified byte-for-byte).

MASTER node keystore:

- Enterprise Root CA G2: <SHA256 verified>

- Public Root CA (DigiCert Global Root G2): <SHA256 verified>

QUERY node truststore :

- enterprise-root-g2: <matches keystore> [Matches]

- digicert-root-g2: <matches keystore> [Matches]

The truststore content is verified correct.

Root Cause - Extended Key Usage:

QUERY node logs:

[2026-05-04T12:13:55,225][WARN ][o.e.c.s.DiagnosticTrustManager] [QUERY00000C] failed to establish trust with client at [<unknown host>]; the client provided a certificate with subject name [CN=node.example.com,O=Enterprise Corp,L=City,ST=State,C=US], fingerprint [<redacted>], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth]; the certificate is valid between [2026-04-10T14:35:57Z] and [2026-07-19T13:35:57Z] (current time is [2026-05-04T12:13:55.225233900Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate is issued by [CN=Enterprise TLS CA G2,O=CA Provider,C=US]; the certificate is signed by (subject [CN=Enterprise TLS CA G2,O=CA Provider,C=US] fingerprint [<redacted>]) signed by (subject [CN=Enterprise Root CA G2,O=CA Provider,C=US] fingerprint [<redacted>] {trusted issuer}) signed by (subject [CN=DigiCert Global Root G2,OU=www.digicert.com,O=DigiCert Inc,C=US] fingerprint [<redacted>] {trusted issuer}) which is self-issued; the [CN=DigiCert Global Root G2,OU=www.digicert.com,O=DigiCert Inc,C=US] certificate is trusted in this ssl context ([xpack.security.transport.ssl (with trust configuration: StoreTrustConfig{path=certs/transport-truststore.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})])
sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication

MASTER node logs:

[2026-05-04T11:59:47,540][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [MASTER000010] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.0.0.4:49558, remoteAddress=10.0.0.12/10.0.0.12:9300, profile=default}
[2026-05-04T11:59:47,543][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [MASTER000010] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.0.0.4:49560, remoteAddress=10.0.0.12/10.0.0.12:9300, profile=default}
[2026-05-04T11:59:47,545][WARN ][o.e.c.c.Coordinator      ] [MASTER000010] received join request from [{QUERY00000C}{<node-id>}{<ephemeral-id>}{QUERY00000C}{10.0.0.12}{10.0.0.12:9300}{8.14.1}{<attributes>}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [QUERY00000C][10.0.0.12:9300] general node connection failure
    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.lambda$onResponse$2(TcpTransport.java:1132) ~[elasticsearch-8.14.1.jar:?]
    at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.14.1.jar:?]

client did not trust this server's certificate

Analysis:

  • All node certificates have extendedKeyUsage: [serverAuth] but lack clientAuth
  • Elasticsearch transport layer uses mutual TLS requiring both server and client authentication
  • With keystore-only configuration: these certificates work fine (lenient validation)
  • With truststore: strict Extended Key Usage validation fails in both directions

Question:
Does truststore configuration enforce stricter Extended Key Usage validation than keystore-only mode? If so, does this mean all node certificates must be reissued with both serverAuth and clientAuth before deploying explicit truststore in production?

This creates a challenge: We need truststore for zero-downtime certificate rotation, but deploying truststore requires reissuing all certificates first (which requires downtime).

Is there a configuration option to relax Extended Key Usage validation when using explicit truststore, or is certificate reissuance the only path forward?