Failed to initialize SSL TrustManager

ES Version 7.12
AKS: 1:20.7

Certificate passwords added elastic keystore and value verified

xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

xpack settings

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.supported_protocols: TLSv1.2    
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/chain-test-rp-search.pfx
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/chain-test-rp-search.pfx
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.client_authentication: optional
xpack.security.http.ssl.verification_mode: certificate    
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/chain-test-rp-search-data.pfx
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/chain-test-rp-search-data.pfx

Log Trace

ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.];
Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2103)
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:220)
        at java.base/java.security.KeyStore.load(KeyStore.java:1472)
        at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:98)
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:66)
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439)
        at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1224)
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528)
        at java.base/java.util.HashMap.forEach(HashMap.java:1425)
        at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521)
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526)
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144)
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:462)
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:292)
        at org.elasticsearch.node.Node.lambda$new$17(Node.java:567)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
        at org.elasticsearch.node.Node.<init>(Node.java:571)
        at org.elasticsearch.node.Node.<init>(Node.java:278)
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)

The only plausible reason for seeing that error message is that your xpack.security.transport.ssl.truststore.secure_password value isn't correct.

I know you said you verified it, but how did you verify it? Is it possible you had a typo while entering it into the keystore?
Or is it possible that it's been added to the wrong keystore (if you have multiple installs of Elasticsearch)?

There are multiple issues

Issue #1

  • We generated selfsigned password protected PFX file (with full chain).
  • Verified the chain with below command
keytool -list -keystone fullChainCert.pfx -storetype pkcs12 -v
  • We uploaded the PFX file to Azure Portal, it removes the password
  • So, essentially our PFX file becomes password free
  • This is the root cause of the this error, trying to supply password for PFX file that wasn't protected with password

Issue #2

  • We automatically sync PFX file using CSI driver and ManagedIdentity
  • We tried our luck by removing passwords in elastisearch.yaml file. That didn't work, we were getting Empty certificate chain error
  • Not sure where issue was, it could be the Elasticsearch, Azure KeyVault or both
  • One thing we noticed was Azure CSi driver sync'ed PFX file doe not pass verification
keytool -list -keystone fullChainCert.pfx -storetype pkcs12 -v

How we resolved

  • Encrypted PFX private key with AES256 alg
  • Created a private Elasticsearch Image that has all PEM files (rootCA, IssueingCA, cert and encrypted key) located in /usr/share/Elasticsearch/config/certs folder
  • Synced ASE decryption key from Azure using CSI driver
  • We did not like this solution but our PROD has 200 machines & we need an automated way to enable TLS/SSL

If anyone has a better way let us know!!!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.