Failed to initialize SSL TrustManager

Elastic Stack Elasticsearch
1

ES Version 7.12
AKS: 1:20.7

Certificate passwords added elastic keystore and value verified

xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

xpack settings

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.supported_protocols: TLSv1.2    
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/chain-test-rp-search.pfx
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/chain-test-rp-search.pfx
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.client_authentication: optional
xpack.security.http.ssl.verification_mode: certificate    
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/chain-test-rp-search-data.pfx
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/chain-test-rp-search-data.pfx

Log Trace

ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.];
Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2103)
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:220)
        at java.base/java.security.KeyStore.load(KeyStore.java:1472)
        at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:98)
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:66)
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439)
        at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1224)
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528)
        at java.base/java.util.HashMap.forEach(HashMap.java:1425)
        at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521)
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526)
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144)
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:462)
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:292)
        at org.elasticsearch.node.Node.lambda$new$17(Node.java:567)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
        at org.elasticsearch.node.Node.<init>(Node.java:571)
        at org.elasticsearch.node.Node.<init>(Node.java:278)
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
2

The only plausible reason for seeing that error message is that your xpack.security.transport.ssl.truststore.secure_password value isn't correct.

I know you said you verified it, but how did you verify it? Is it possible you had a typo while entering it into the keystore?
Or is it possible that it's been added to the wrong keystore (if you have multiple installs of Elasticsearch)?

3

There are multiple issues

Issue #1

  • We generated selfsigned password protected PFX file (with full chain).
  • Verified the chain with below command
keytool -list -keystone fullChainCert.pfx -storetype pkcs12 -v
  • We uploaded the PFX file to Azure Portal, it removes the password
  • So, essentially our PFX file becomes password free
  • This is the root cause of the this error, trying to supply password for PFX file that wasn't protected with password

Issue #2

  • We automatically sync PFX file using CSI driver and ManagedIdentity
  • We tried our luck by removing passwords in elastisearch.yaml file. That didn't work, we were getting Empty certificate chain error
  • Not sure where issue was, it could be the Elasticsearch, Azure KeyVault or both
  • One thing we noticed was Azure CSi driver sync'ed PFX file doe not pass verification
keytool -list -keystone fullChainCert.pfx -storetype pkcs12 -v

How we resolved

  • Encrypted PFX private key with AES256 alg
  • Created a private Elasticsearch Image that has all PEM files (rootCA, IssueingCA, cert and encrypted key) located in /usr/share/Elasticsearch/config/certs folder
  • Synced ASE decryption key from Azure using CSI driver
  • We did not like this solution but our PROD has 200 machines & we need an automated way to enable TLS/SSL

If anyone has a better way let us know!!!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.