Hello:
I am running into issues with using BouncyCastle keystores and truststores. The error being returned is:
Error finalising cipher data: mac check in CCM failed.
Any help with understanding this error and how to resolve it would be greatly appreciated.
The stores appear to be valid, as I am able to list them using keytool, and changing the password in elasticsearch.yml results in a different error.
(Note: While the ultimate goal is to enable FIPS, the Elastic node does not yet have fips_mode.enabled set to true as yet, I'm just taking it one step at a time.)
Summary of steps taken so far (details are further down):
- Stores converted to BCFKS
- java.security updated
- java.policy updated
- elasticsearch.yml updated
Full error:
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL KeyManager]; nested: UnrecoverableKeyException[BCFKS KeyStore unable to recover private key (instance): Error finalising cipher data: mac check in CCM failed];
Conversion to BCFKS
Here's the command used to convert the existing http.p12 to BCFKS:
keytool -importkeystore -srckeystore http.p12 -srcstoretype pkcs12 -srcstorepass <snip> -destkeystore http.bcfks -deststoretype BCFKS -providername BCFIPS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /usr/share/elasticsearch/lib/bc-fips-1.0.2.3.jar
Reading the file with keytool appears to confirm it worked:
keytool -keystore http.bcfks -storetype BCFKS -storepass <snip> -providerpath /usr/share/elasticsearch/lib/bc-fips-1.0.2.3.jar -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -list
Keystore type: BCFKS
Keystore provider: BCFIPS
Your keystore contains 2 entries
ca, Aug 26, 2022, trustedCertEntry,
Certificate fingerprint (SHA-256): <snip>
http, Aug 26, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): <snip>
Here's the command used to convert the existing elastic-certificates.p12 to BCFKS:
keytool -importkeystore -srckeystore elastic-certificates.p12 -srcstoretype pkcs12 -srcstorepass <snip> -destkeystore elastic-certificates.bcfks -deststoretype BCFKS -providername BCFIPS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /usr/share/elasticsearch/lib/bc-fips-1.0.2.3.jar
Reading the file with keytool appears to confirm it worked:
keytool -keystore elastic-certificates.bcfks -storetype BCFKS -storepass <snip> -providerpath /usr/share/elasticsearch/lib/bc-fips-1.0.2.3.jar -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -list
Keystore type: BCFKS
Keystore provider: BCFIPS
Your keystore contains 2 entries
ca, Aug 26, 2022, trustedCertEntry,
Certificate fingerprint (SHA-256): <snip>
instance, Aug 26, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA-256): <snip>
The following changes were made to elasticsearch.yml:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.bcfks
xpack.security.transport.ssl.keystore.type: BCFKS
xpack.security.transport.ssl.keystore.password: <snip>
xpack.security.transport.ssl.truststore.path: elastic-certificates.bcfks
xpack.security.transport.ssl.truststore.type: BCFKS
xpack.security.transport.ssl.truststore.password: <snip>
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.bcfks
xpack.security.http.ssl.keystore.type: BCFKS
xpack.security.http.ssl.keystore.password: <snip>
java.security now includes this:
security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:DEFRND[HmacSHA512];ENABLE{ALL};
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS crypto.policy=unlimited
security.provider.3=SUN
security.provider.4=SunRsaSign
security.provider.5=SunEC
security.provider.6=SunJSSE
security.provider.7=SunJCE
security.provider.8=SunJGSS
security.provider.9=SunSASL
security.provider.10=XMLDSig
security.provider.11=SunPCSC
security.provider.12=JdkLDAP
security.provider.13=JdkSASL
java.policy includes this:
// default permissions granted to all domains
grant {
// allows anyone to listen on dynamic ports
permission java.net.SocketPermission "localhost:0", "listen";
// "standard" properies that can be read by anyone
permission java.util.PropertyPermission "java.version", "read";
permission java.util.PropertyPermission "java.vendor", "read";
permission java.util.PropertyPermission "java.vendor.url", "read";
permission java.util.PropertyPermission "java.class.version", "read";
permission java.util.PropertyPermission "os.name", "read";
permission java.util.PropertyPermission "os.version", "read";
permission java.util.PropertyPermission "os.arch", "read";
permission java.util.PropertyPermission "file.separator", "read";
permission java.util.PropertyPermission "path.separator", "read";
permission java.util.PropertyPermission "line.separator", "read";
permission java.util.PropertyPermission
"java.specification.version", "read";
permission java.util.PropertyPermission "java.specification.vendor", "read";
permission java.util.PropertyPermission "java.specification.name", "read";
permission java.util.PropertyPermission
"java.vm.specification.version", "read";
permission java.util.PropertyPermission
"java.vm.specification.vendor", "read";
permission java.util.PropertyPermission
"java.vm.specification.name", "read";
permission java.util.PropertyPermission "java.vm.version", "read";
permission java.util.PropertyPermission "java.vm.vendor", "read";
permission java.util.PropertyPermission "java.vm.name", "read";
// policy which allows everything to everyone
permission java.security.AllPermission "", "";
};
Log snippet:
[2022-08-30T21:37:08,059][ERROR][o.e.b.Bootstrap ] [<snip>] Exception
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:548) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:544) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:145) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:525) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:338) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:736) ~[elasticsearch-7.17.1.jar:7.17.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:750) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.17.1.jar:7.17.1]
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL KeyManager
at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createKeyManager(StoreKeyConfig.java:93) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:454) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:546) ~[?:?]
... 26 more
Caused by: java.security.UnrecoverableKeyException: BCFKS KeyStore unable to recover private key (instance): Error finalising cipher data: mac check in CCM failed
at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineGetKey(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]
at java.security.KeyStore.getKey(KeyStore.java:1050) ~[?:?]
at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:141) ~[?:?]
at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:64) ~[?:?]
at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:275) ~[?:?]
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.keyManager(CertParsingUtils.java:199) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createKeyManager(StoreKeyConfig.java:85) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:454) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:546) ~[?:?]
... 26 more
[2022-08-30T21:37:08,067][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [<snip>] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL KeyManager]; nested: UnrecoverableKeyException[BCFKS KeyStore unable to recover private key (instance): Error finalising cipher data: mac check in CCM failed];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.1.jar:7.17.1]
at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.1.jar:7.17.1]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:548) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:544) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:145) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:525) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:338) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:736) ~[elasticsearch-7.17.1.jar:7.17.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:750) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.1.jar:7.17.1]
... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL KeyManager
at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createKeyManager(StoreKeyConfig.java:93) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:454) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:546) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:544) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:145) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:525) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:338) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:736) ~[elasticsearch-7.17.1.jar:7.17.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:750) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.1.jar:7.17.1]
... 6 more
Caused by: java.security.UnrecoverableKeyException: BCFKS KeyStore unable to recover private key (instance): Error finalising cipher data: mac check in CCM failed
at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineGetKey(Unknown Source) ~[bc-fips-1.0.2.3.jar:1.0.2.3]
at java.security.KeyStore.getKey(KeyStore.java:1050) ~[?:?]
at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:141) ~[?:?]
at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:64) ~[?:?]
at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:275) ~[?:?]
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.keyManager(CertParsingUtils.java:199) ~[?:?]
at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createKeyManager(StoreKeyConfig.java:85) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:454) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:546) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1421) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:544) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:145) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:525) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:338) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:736) ~[elasticsearch-7.17.1.jar:7.17.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:750) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.1.jar:7.17.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.1.jar:7.17.1]
... 6 more
Thanks in advance!