ElasticSearch FIPS (BouncyCastle)

I am tasked with trying to get Elasticsearch working in FIPS mode. I know there is probably no experts on this topic in the forums but wanted to see if anyone could give me just a working solution.

In my Elasticsearch.yml i have.

## FIPS
xpack.security.fips_mode.enabled: true
xpack.security.authc.password_hashing.algorithm: pbkdf2_stretch

# SSL Stuff
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.keystore.password: {12 character password}
xpack.security.http.ssl.truststore.password: {12 character password}
xpack.security.http.ssl.keystore.type: BCFKS
xpack.security.http.ssl.truststore.type: BCFKS
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/node.bcfks
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/node.bcfks
xpack.security.http.ssl.supported_protocols: TLSv1, TLSv1.1, TLSv1.2, TLSv1.3

I am using the embedded JAVA JVM included with Elasticsearch 7.15.1, and I modified the conf/security/java.security as follows:

security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:DEFRND[HmacSHA512];ENABLE{ALL};
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS crypto.policy=unlimited
security.provider.3=SUN
security.provider.4=SunRsaSign
security.provider.5=SunEC
security.provider.6=SunJSSE
security.provider.7=SunJCE
security.provider.8=SunJGSS
security.provider.9=SunSASL
security.provider.10=XMLDSig
security.provider.11=SunPCSC
security.provider.12=JdkLDAP
security.provider.13=JdkSASL
security.provider.14=SunPKCS11

keystore.type=BCFKS
keystore.type.compat=false

Nothing else was modified from the original java.security.

As of right now i get the JAVA error when trying to start Elasticsearch

[2022-04-26T14:09:38,281][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]
	at java.util.HashMap.forEach(HashMap.java:1420) ~[?:?]
	at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.1.jar:7.15.1]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) [elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) [elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158) [elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114) [elasticsearch-cli-7.15.1.jar:7.15.1]
	at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123) [elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.15.1.jar:7.15.1]
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:51) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1219) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
	... 26 more
Caused by: java.security.KeyStoreException: problem accessing trust store
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:297) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:49) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1219) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
	... 26 more
Caused by: java.io.IOException: DER length more than 4 bytes: 109
	at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source) ~[bc-fips-1.0.2.jar:?]
	at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source) ~[bc-fips-1.0.2.jar:?]
	at org.bouncycastle.asn1.ASN1InputStream.readObject(Unknown Source) ~[bc-fips-1.0.2.jar:?]
	at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(Unknown Source) ~[bc-fips-1.0.2.jar:?]
	at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]
	at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390) ~[?:?]
	at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336) ~[?:?]
	at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57) ~[?:?]
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:297) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:49) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1219) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
	... 26 more
[2022-04-26T14:09:38,286][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize a TrustManagerFactory]; nested: KeyStoreException[problem accessing trust store]; nested: IOException[DER length more than 4 bytes: 109];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:171) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114) ~[elasticsearch-cli-7.15.1.jar:7.15.1]
	at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.15.1.jar:7.15.1]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]
	at java.util.HashMap.forEach(HashMap.java:1420) ~[?:?]
	at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.1.jar:7.15.1]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-7.15.1.jar:7.15.1]
	... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:51) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1219) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
	at java.util.HashMap.forEach(HashMap.java:1420) ~[?:?]
	at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.1.jar:7.15.1]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-7.15.1.jar:7.15.1]
	... 6 more
Caused by: java.security.KeyStoreException: problem accessing trust store
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:297) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:49) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1219) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
	at java.util.HashMap.forEach(HashMap.java:1420) ~[?:?]
	at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.1.jar:7.15.1]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-7.15.1.jar:7.15.1]
	... 6 more
Caused by: java.io.IOException: DER length more than 4 bytes: 109
	at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source) ~[bc-fips-1.0.2.jar:?]
	at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source) ~[bc-fips-1.0.2.jar:?]
	at org.bouncycastle.asn1.ASN1InputStream.readObject(Unknown Source) ~[bc-fips-1.0.2.jar:?]
	at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(Unknown Source) ~[bc-fips-1.0.2.jar:?]
	at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]
	at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390) ~[?:?]
	at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336) ~[?:?]
	at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57) ~[?:?]
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:297) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:49) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1219) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
	at java.util.HashMap.forEach(HashMap.java:1420) ~[?:?]
	at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.1.jar:7.15.1]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.15.1.jar:7.15.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-7.15.1.jar:7.15.1]
	... 6 more

It makes me think that maybe something with my conversion of the BCKFS went wrong, or i'm missing something in the java.security? Is there a known GOOD (java.security) file that should be used for FIPS and BCKFS/BCFIPS?

Also guidance here would be appreciated. We are just trying to get FIPS complaint, not really needing to certify it.

Hi there

As you said yourself, it's hard to find any FIPS 140-2 experts in the forums and I encourage you to seek for assistance from an appropriate partner.

  1. You would need to change your security policy file (java.policy) too. See Appendix B in https://downloads.bouncycastle.org/fips-java/BC-FJA-UserGuide-1.0.2.pdf

  2. How did you create the node.bcfks keystores ?

I have tried multiple ways to create the BCFKS.

Method #1: Tried to convert PKCS12 to BCFKS using:

keytool -importkeystore -srckeystore elastic-certificates.p12 -srcstorepass XXXXXX  -srcstoretype pkcs12 -destkeystore nodes.bcfks -deststorepass XXXXXXXXXXXXXXX -deststoretype bcfks -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /usr/share/elasticsearch/jdk/lib/bc-fips-1.0.2.jar

I even tried a command like this following the steps from:
https://docs.datafabric.hpe.com/70/ClusterAdministration/admin/cluster/UsingJavaKeytoolWithBCFKS.html

keytool -keystore ssl_keystore.bcfks -storetype BCFKS -providername BCFIPS -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /usr/share/elasticsearch/jdk/lib/bc-fips-1.0.2.jar -alias node1.example.com -genkeypair -sigalg SHA512withRSA -keyalg RSA -storepass XXXXXXXXXXXXX -dname CN=node1.example.com -keypass XXXXXXXXXXXXX 

I get the same results no matter how i try to use the store. Am I doing something wrong here?

No, that looks fine to me. I’d focus on fixing the policy

this is basically my JAVA policy, from hat I can tell there is nothing wrong with it, i get no errors about permission issues when it tries to boot, but instantly dies with that strange DER message.

// default permissions granted to all domains
grant {


    permission java.security.SecurityPermission "putProviderProperty.BCFIPS";
    permission java.security.SecurityPermission "putProviderProperty.BCJSSE";
    permission java.security.SecurityPermission "getProperty.jdk.tls.disabledAlgorithms";
    permission java.security.SecurityPermission "getProperty.jdk.certpath.disabledAlgorithms";
    permission java.security.SecurityPermission "getProperty.keystore.type.compat";
    permission java.lang.RuntimePermission "getProtectionDomain";
    permission java.util.PropertyPermission "java.runtime.name", "read";
    permission org.bouncycastle.crypto.CryptoServicesPermission "tlsAlgorithmsEnabled";
    //io.netty.handler.codec.DecoderException
    permission java.lang.RuntimePermission "accessClassInPackage.sun.security.internal.spec";
    //java.security.InvalidAlgorithmParameterException: Cannot process GCMParameterSpec
    permission java.lang.RuntimePermission "accessDeclaredMembers";
    permission java.util.PropertyPermission "intellij.debug.agent", "read";
    permission java.util.PropertyPermission "intellij.debug.agent", "write";
    permission org.bouncycastle.crypto.CryptoServicesPermission "exportSecretKey";
    permission org.bouncycastle.crypto.CryptoServicesPermission "exportPrivateKey";
    //permission org.bouncycastle.crypto.CryptoServicesPermission "unapprovedModeEnabled", "Y";
    //permission org.bouncycastle.crypto.CryptoServicesPermission "defaultRandomConfig", "Y";
    permission java.io.FilePermission "/etc/elasticsearch/mybcfks.bcfks", "read";
    permission java.io.FilePermission "/etc/elasticsearch/ssl_keystore.bcfks", "read";
    permission java.io.FilePermission "/etc/elasticsearch/ssl_truststore.bcfks", "read";
    permission java.io.FilePermission "/usr/share/elasticsearch/jdk/lib/security/jssecacerts", "read";
    permission java.io.FilePermission "/usr/share/elasticsearch/jdk/lib/security/cacerts", "read";


    // allows anyone to listen on dynamic ports
    permission java.net.SocketPermission "localhost:0", "listen";

    // "standard" properies that can be read by anyone
    permission java.util.PropertyPermission "java.version", "read";
    permission java.util.PropertyPermission "java.vendor", "read";
    permission java.util.PropertyPermission "java.vendor.url", "read";
    permission java.util.PropertyPermission "java.class.version", "read";
    permission java.util.PropertyPermission "os.name", "read";
    permission java.util.PropertyPermission "os.version", "read";
    permission java.util.PropertyPermission "os.arch", "read";
    permission java.util.PropertyPermission "file.separator", "read";
    permission java.util.PropertyPermission "path.separator", "read";
    permission java.util.PropertyPermission "line.separator", "read";
    permission java.util.PropertyPermission
                   "java.specification.version", "read";
    permission java.util.PropertyPermission "java.specification.vendor", "read";
    permission java.util.PropertyPermission "java.specification.name", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.version", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.vendor", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.name", "read";
    permission java.util.PropertyPermission "java.vm.version", "read";
    permission java.util.PropertyPermission "java.vm.vendor", "read";
    permission java.util.PropertyPermission "java.vm.name", "read";


};

Here is the full boot message I get with errors, you can see the Messages about it picking up variables and settings. I am stumped..

[2022-04-27T18:55:09,715][INFO ][o.e.n.Node               ] [node-1] version[7.15.1], pid[26478], build[default/rpm/83c34f456ae29d60e94d886e455e6a3409bba9ed/2021-10-07T21:56:19.031608185Z], OS[Linux/5.4.17-2136.306.1.3.el7uek.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17/17+35]
[2022-04-27T18:55:09,733][INFO ][o.e.n.Node               ] [node-1] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-04-27T18:55:09,734][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms4g, -Xmx4g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9792561885369746183, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2147483648, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2022-04-27T18:55:10,551][WARN ][stderr                   ] [node-1] Apr 27, 2022 6:55:10 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
[2022-04-27T18:55:10,551][WARN ][stderr                   ] [node-1] INFO: Found string security property [jdk.tls.disabledAlgorithms]: SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL
[2022-04-27T18:55:10,552][WARN ][stderr                   ] [node-1] Apr 27, 2022 6:55:10 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
[2022-04-27T18:55:10,552][WARN ][stderr                   ] [node-1] INFO: Found string security property [jdk.certpath.disabledAlgorithms]: MD2, RSA keySize < 1024
[2022-04-27T18:55:14,793][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [aggs-matrix-stats]
[2022-04-27T18:55:14,793][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [analysis-common]
[2022-04-27T18:55:14,794][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [constant-keyword]
[2022-04-27T18:55:14,794][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [frozen-indices]
[2022-04-27T18:55:14,794][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-common]
[2022-04-27T18:55:14,794][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-geoip]
[2022-04-27T18:55:14,794][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-user-agent]
[2022-04-27T18:55:14,795][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [kibana]
[2022-04-27T18:55:14,795][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-expression]
[2022-04-27T18:55:14,795][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-mustache]
[2022-04-27T18:55:14,795][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-painless]
[2022-04-27T18:55:14,795][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-extras]
[2022-04-27T18:55:14,796][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-version]
[2022-04-27T18:55:14,796][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [parent-join]
[2022-04-27T18:55:14,796][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [percolator]
[2022-04-27T18:55:14,796][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [rank-eval]
[2022-04-27T18:55:14,796][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [reindex]
[2022-04-27T18:55:14,796][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repositories-metering-api]
[2022-04-27T18:55:14,797][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-encrypted]
[2022-04-27T18:55:14,797][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-url]
[2022-04-27T18:55:14,797][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [runtime-fields-common]
[2022-04-27T18:55:14,797][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [search-business-rules]
[2022-04-27T18:55:14,797][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [searchable-snapshots]
[2022-04-27T18:55:14,816][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [snapshot-repo-test-kit]
[2022-04-27T18:55:14,816][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [spatial]
[2022-04-27T18:55:14,816][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [systemd]
[2022-04-27T18:55:14,817][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transform]
[2022-04-27T18:55:14,817][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transport-netty4]
[2022-04-27T18:55:14,817][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [unsigned-long]
[2022-04-27T18:55:14,817][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [vector-tile]
[2022-04-27T18:55:14,817][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [vectors]
[2022-04-27T18:55:14,818][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [wildcard]
[2022-04-27T18:55:14,818][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-aggregate-metric]
[2022-04-27T18:55:14,818][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-analytics]
[2022-04-27T18:55:14,818][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-async]
[2022-04-27T18:55:14,818][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-async-search]
[2022-04-27T18:55:14,819][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-autoscaling]
[2022-04-27T18:55:14,819][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ccr]
[2022-04-27T18:55:14,819][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-core]
[2022-04-27T18:55:14,819][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-data-streams]
[2022-04-27T18:55:14,819][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-deprecation]
[2022-04-27T18:55:14,820][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-enrich]
[2022-04-27T18:55:14,820][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-eql]
[2022-04-27T18:55:14,820][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-fleet]
[2022-04-27T18:55:14,820][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-graph]
[2022-04-27T18:55:14,820][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-identity-provider]
[2022-04-27T18:55:14,821][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ilm]
[2022-04-27T18:55:14,821][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-logstash]
[2022-04-27T18:55:14,821][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ml]
[2022-04-27T18:55:14,821][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-monitoring]
[2022-04-27T18:55:14,821][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ql]
[2022-04-27T18:55:14,822][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-rollup]
[2022-04-27T18:55:14,822][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-security]
[2022-04-27T18:55:14,822][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-shutdown]
[2022-04-27T18:55:14,822][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-sql]
[2022-04-27T18:55:14,822][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-stack]
[2022-04-27T18:55:14,823][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-text-structure]
[2022-04-27T18:55:14,823][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-voting-only-node]
[2022-04-27T18:55:14,823][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-watcher]
[2022-04-27T18:55:14,823][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2022-04-27T18:55:14,865][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (/dev/mapper/ol-root)]], net usable_space [29.2gb], net total_space [34.9gb], types [xfs]
[2022-04-27T18:55:14,865][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [4gb], compressed ordinary object pointers [true]
[2022-04-27T18:55:15,019][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [AeEoJJAkSGKOyl-Wolnqjw], cluster name [fips], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-04-27T18:55:21,888][WARN ][stderr                   ] [node-1] Apr 27, 2022 6:55:21 PM org.bouncycastle.jsse.provider.PropertyUtils getBooleanSecurityProperty
[2022-04-27T18:55:21,889][WARN ][stderr                   ] [node-1] INFO: Found boolean security property [keystore.type.compat]: true
[2022-04-27T18:55:21,890][WARN ][stderr                   ] [node-1] Apr 27, 2022 6:55:21 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T18:55:21,890][WARN ][stderr                   ] [node-1] INFO: Found string system property [java.home]: /usr/share/elasticsearch/jdk
[2022-04-27T18:55:21,918][WARN ][stderr                   ] [node-1] Apr 27, 2022 6:55:21 PM org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi engineInit
[2022-04-27T18:55:21,918][WARN ][stderr                   ] [node-1] WARNING: Skipped default trust store
[2022-04-27T18:55:21,918][WARN ][stderr                   ] [node-1] java.io.IOException: DER length more than 4 bytes: 109
[2022-04-27T18:55:21,918][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.asn1.ASN1InputStream.readLength(ASN1InputStream.java:350)
[2022-04-27T18:55:21,918][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.asn1.ASN1InputStream.readLength(ASN1InputStream.java:115)
[2022-04-27T18:55:21,919][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.asn1.ASN1InputStream.readObject(ASN1InputStream.java:240)
[2022-04-27T18:55:21,919][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(ProvBCFKS.java:1226)
[2022-04-27T18:55:21,919][WARN ][stderr                   ] [node-1] 	at java.base/java.security.KeyStore.load(KeyStore.java:1473)
[2022-04-27T18:55:21,919][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.getDefaultTrustStore(ProvTrustManagerFactorySpi.java:112)
[2022-04-27T18:55:21,920][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.engineInit(ProvTrustManagerFactorySpi.java:162)
[2022-04-27T18:55:21,920][WARN ][stderr                   ] [node-1] 	at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282)
[2022-04-27T18:55:21,920][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvSSLContextSpi.selectX509TrustManager(ProvSSLContextSpi.java:925)
[2022-04-27T18:55:21,939][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvSSLContextSpi.engineInit(ProvSSLContextSpi.java:879)
[2022-04-27T18:55:21,940][WARN ][stderr                   ] [node-1] 	at java.base/javax.net.ssl.SSLContext.init(SSLContext.java:314)
[2022-04-27T18:55:21,940][WARN ][stderr                   ] [node-1] 	at org.apache.http.ssl.SSLContexts.createDefault(SSLContexts.java:52)
[2022-04-27T18:55:21,940][WARN ][stderr                   ] [node-1] 	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.getSocketFactory(SSLConnectionSocketFactory.java:194)
[2022-04-27T18:55:21,940][WARN ][stderr                   ] [node-1] 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.getDefaultRegistry(PoolingHttpClientConnectionManager.java:115)
[2022-04-27T18:55:21,940][WARN ][stderr                   ] [node-1] 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:122)
[2022-04-27T18:55:21,941][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.common.blobstore.url.http.URLHttpClient$Factory.<init>(URLHttpClient.java:49)
[2022-04-27T18:55:21,941][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.plugin.repository.url.URLRepositoryPlugin.createComponents(URLRepositoryPlugin.java:83)
[2022-04-27T18:55:21,941][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.node.Node.lambda$new$18(Node.java:622)
[2022-04-27T18:55:21,941][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
[2022-04-27T18:55:21,941][WARN ][stderr                   ] [node-1] 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
[2022-04-27T18:55:21,942][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
[2022-04-27T18:55:21,942][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
[2022-04-27T18:55:21,942][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
[2022-04-27T18:55:21,942][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[2022-04-27T18:55:21,942][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
[2022-04-27T18:55:21,942][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.node.Node.<init>(Node.java:626)
[2022-04-27T18:55:21,943][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.node.Node.<init>(Node.java:288)
[2022-04-27T18:55:21,943][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219)
[2022-04-27T18:55:21,943][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219)
[2022-04-27T18:55:21,943][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399)
[2022-04-27T18:55:21,943][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167)
[2022-04-27T18:55:21,943][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158)
[2022-04-27T18:55:21,944][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
[2022-04-27T18:55:21,944][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
[2022-04-27T18:55:21,944][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.cli.Command.main(Command.java:79)
[2022-04-27T18:55:21,944][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123)
[2022-04-27T18:55:21,945][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
[2022-04-27T18:55:21,945][WARN ][stderr                   ] [node-1] Apr 27, 2022 6:55:21 PM org.bouncycastle.jsse.provider.ProvSSLContextSpi selectX509TrustManager
[2022-04-27T18:55:21,946][WARN ][stderr                   ] [node-1] WARNING: Failed to load default trust managers
[2022-04-27T18:55:21,946][WARN ][stderr                   ] [node-1] java.security.KeyStoreException: Failed to load default trust store
[2022-04-27T18:55:21,946][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.engineInit(ProvTrustManagerFactorySpi.java:182)
[2022-04-27T18:55:21,946][WARN ][stderr                   ] [node-1] 	at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282)
[2022-04-27T18:55:21,946][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvSSLContextSpi.selectX509TrustManager(ProvSSLContextSpi.java:925)
[2022-04-27T18:55:21,946][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvSSLContextSpi.engineInit(ProvSSLContextSpi.java:879)
[2022-04-27T18:55:21,947][WARN ][stderr                   ] [node-1] 	at java.base/javax.net.ssl.SSLContext.init(SSLContext.java:314)
[2022-04-27T18:55:21,947][WARN ][stderr                   ] [node-1] 	at org.apache.http.ssl.SSLContexts.createDefault(SSLContexts.java:52)
[2022-04-27T18:55:21,947][WARN ][stderr                   ] [node-1] 	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.getSocketFactory(SSLConnectionSocketFactory.java:194)
[2022-04-27T18:55:21,947][WARN ][stderr                   ] [node-1] 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.getDefaultRegistry(PoolingHttpClientConnectionManager.java:115)
[2022-04-27T18:55:21,947][WARN ][stderr                   ] [node-1] 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:122)
[2022-04-27T18:55:21,948][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.common.blobstore.url.http.URLHttpClient$Factory.<init>(URLHttpClient.java:49)
[2022-04-27T18:55:21,948][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.plugin.repository.url.URLRepositoryPlugin.createComponents(URLRepositoryPlugin.java:83)
[2022-04-27T18:55:21,948][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.node.Node.lambda$new$18(Node.java:622)
[2022-04-27T18:55:21,948][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
[2022-04-27T18:55:21,948][WARN ][stderr                   ] [node-1] 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
[2022-04-27T18:55:21,948][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
[2022-04-27T18:55:21,949][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
[2022-04-27T18:55:21,949][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
[2022-04-27T18:55:21,949][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[2022-04-27T18:55:21,949][WARN ][stderr                   ] [node-1] 	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
[2022-04-27T18:55:21,949][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.node.Node.<init>(Node.java:626)
[2022-04-27T18:55:21,950][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.node.Node.<init>(Node.java:288)
[2022-04-27T18:55:21,950][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219)
[2022-04-27T18:55:21,950][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219)
[2022-04-27T18:55:21,950][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399)
[2022-04-27T18:55:21,950][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167)
[2022-04-27T18:55:21,951][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158)
[2022-04-27T18:55:21,951][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
[2022-04-27T18:55:21,951][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
[2022-04-27T18:55:21,975][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.cli.Command.main(Command.java:79)
[2022-04-27T18:55:21,976][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123)
[2022-04-27T18:55:21,976][WARN ][stderr                   ] [node-1] 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
[2022-04-27T18:55:21,976][WARN ][stderr                   ] [node-1] Caused by: java.io.IOException: DER length more than 4 bytes: 109
[2022-04-27T18:55:21,976][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.asn1.ASN1InputStream.readLength(ASN1InputStream.java:350)
[2022-04-27T18:55:21,976][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.asn1.ASN1InputStream.readLength(ASN1InputStream.java:115)
[2022-04-27T18:55:21,977][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.asn1.ASN1InputStream.readObject(ASN1InputStream.java:240)
[2022-04-27T18:55:21,977][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(ProvBCFKS.java:1226)
[2022-04-27T18:55:21,977][WARN ][stderr                   ] [node-1] 	at java.base/java.security.KeyStore.load(KeyStore.java:1473)
[2022-04-27T18:55:21,977][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.getDefaultTrustStore(ProvTrustManagerFactorySpi.java:112)
[2022-04-27T18:55:21,977][WARN ][stderr                   ] [node-1] 	at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.engineInit(ProvTrustManagerFactorySpi.java:162)
[2022-04-27T18:55:21,977][WARN ][stderr                   ] [node-1] 	... 30 more

I can't comment on that but I trust you followed the link I shared with you and applied the needed policy changes.

[2022-04-27T18:55:21,919][WARN ][stderr ] [node-1] at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.getDefaultTrustStore(ProvTrustManagerFactorySpi.java:112)

It might be that you need to also convert the default truststore ( cacerts ) that comes with the JDK to BCFKS and set a password for it and then load it explicitly by passing

-Djavax.net.ssl.trustStore=path/to/cacerts.bcfks -Djavax.net.ssl.keyStore=path/to/cacerts.bcfks -Djavax.net.ssl.keyStoreType=BCFKS -Djavax.net.ssl.trustStorePassword=thepasswordyouset -Djavax.net.ssl.keyStorePassword=thepasswordyouset

as extra JVM options
Which also reminds me that you need to pass -Dorg.bouncycastle.fips.approved_only=true too as a JVM arg

Finally, please note that we take no responsibility for the correctness of this configuration and whether or not this leads to your environment being compliant and I still strongly suggest that you speak with someone that is an expert on the matter.

Interesting findings so far today, if i enable that option: -Dorg.bouncycastle.fips.approved_only=true
It dies really quick with:

Apr 27 23:25:43 node1 systemd-entrypoint[2604]: WARNING: A terminally deprecated method in java.lang.System has been called
Apr 27 23:25:43 node1 systemd-entrypoint[2604]: WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Elasticsearch (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.1.j
Apr 27 23:25:43 node1 systemd-entrypoint[2604]: WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Elasticsearch
Apr 27 23:25:43 node1 systemd-entrypoint[2604]: WARNING: System::setSecurityManager will be removed in a future release
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.security.GeneralSecurityException: Error generating an encryption key from
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: Likely root cause: org.bouncycastle.crypto.fips.FipsUnapprovedOperationError: password must be at least 112 bits
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.bouncycastle.crypto.fips.FipsPBKD$Parameters.<init>(FipsPBKD.java:90)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.bouncycastle.crypto.fips.FipsPBKD$Parameters.<init>(FipsPBKD.java:63)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.bouncycastle.crypto.fips.FipsPBKD$ParametersBuilder.using(FipsPBKD.java:56)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.bouncycastle.jcajce.provider.ProvPBEPBKDF2$BasePBKDF2.engineGenerateSecret(ProvPBEPBKDF2.java:250)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at java.base/javax.crypto.SecretKeyFactory.generateSecret(SecretKeyFactory.java:340)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.common.settings.KeyStoreWrapper.createCipher(KeyStoreWrapper.java:305)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.common.settings.KeyStoreWrapper.decrypt(KeyStoreWrapper.java:358)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:258)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:230)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:345)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.cli.Command.main(Command.java:79)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123)
Apr 27 23:25:46 node1 systemd-entrypoint[2604]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Apr 27 23:25:46 node1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE

Not sure why it gives that message, the password is being set via ES_JAVA_OPTIONS and i confirmed that it shows when I do 'ps ax | grep java'.

If i remove that option it does now start but TLS is in a very strange place:

2022-04-27T23:22:19,015][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (/dev/mapper/ol-root)]], net usable_space [29.2gb], net total_space [34.9gb], types [xfs]
[2022-04-27T23:22:19,016][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [4gb], compressed ordinary object pointers [true]
[2022-04-27T23:22:19,150][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [AeEoJJAkSGKOyl-Wolnqjw], cluster name [fips], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-04-27T23:22:26,978][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:26 PM org.bouncycastle.jsse.provider.PropertyUtils getBooleanSecurityProperty
[2022-04-27T23:22:26,978][WARN ][stderr                   ] [node-1] INFO: Found boolean security property [keystore.type.compat]: true
[2022-04-27T23:22:26,979][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:26 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T23:22:27,000][WARN ][stderr                   ] [node-1] INFO: Found string system property [javax.net.ssl.trustStore]: /usr/share/elasticsearch/jdk/lib/security/cacerts.bcfks
[2022-04-27T23:22:27,001][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:27 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T23:22:27,001][WARN ][stderr                   ] [node-1] INFO: Found string system property [javax.net.ssl.trustStoreType]: BCFKS
[2022-04-27T23:22:27,026][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:27 PM org.bouncycastle.jsse.provider.PropertyUtils getSensitiveStringSystemProperty
[2022-04-27T23:22:27,026][WARN ][stderr                   ] [node-1] INFO: Found sensitive string system property [javax.net.ssl.trustStorePassword]
[2022-04-27T23:22:28,180][INFO ][o.e.x.c.s.SSLService     ] [node-1] diagnostic messages for SSL/TLS trust failures are not enabled in FIPS 140 mode by default.
[2022-04-27T23:22:28,577][ERROR][o.e.x.c.s.SSLService     ] [node-1] unsupported ciphers [[TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256]] were requested but cannot be used in this JVM, however there are supported ciphers that will be used [[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]]. If you are trying to use ciphers with a key length greater than 128 bits on an Oracle JVM, you will need to install the unlimited strength JCE policy files.
[2022-04-27T23:22:28,578][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:28 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T23:22:28,578][WARN ][stderr                   ] [node-1] INFO: Found string system property [javax.net.ssl.trustStore]: /usr/share/elasticsearch/jdk/lib/security/cacerts.bcfks
[2022-04-27T23:22:28,579][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:28 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T23:22:28,579][WARN ][stderr                   ] [node-1] INFO: Found string system property [javax.net.ssl.trustStoreType]: BCFKS
[2022-04-27T23:22:28,579][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:28 PM org.bouncycastle.jsse.provider.PropertyUtils getSensitiveStringSystemProperty
[2022-04-27T23:22:28,579][WARN ][stderr                   ] [node-1] INFO: Found sensitive string system property [javax.net.ssl.trustStorePassword]
[2022-04-27T23:22:28,740][INFO ][o.e.x.c.s.SSLService     ] [node-1] diagnostic messages for SSL/TLS trust failures are not enabled in FIPS 140 mode by default.
[2022-04-27T23:22:28,953][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/2515] [Main.cc@122] controller (64 bit): Version 7.15.1 (Build 96c59930f1bbe9) Copyright (c) 2021 Elasticsearch BV
[2022-04-27T23:22:29,070][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:29 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T23:22:29,071][WARN ][stderr                   ] [node-1] INFO: Found string system property [javax.net.ssl.trustStore]: /usr/share/elasticsearch/jdk/lib/security/cacerts.bcfks
[2022-04-27T23:22:29,072][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:29 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T23:22:29,072][WARN ][stderr                   ] [node-1] INFO: Found string system property [javax.net.ssl.trustStoreType]: BCFKS
[2022-04-27T23:22:29,072][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:29 PM org.bouncycastle.jsse.provider.PropertyUtils getSensitiveStringSystemProperty
[2022-04-27T23:22:29,073][WARN ][stderr                   ] [node-1] INFO: Found sensitive string system property [javax.net.ssl.trustStorePassword]
[2022-04-27T23:22:29,609][INFO ][o.e.x.s.a.Realms         ] [node-1] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-04-27T23:22:29,626][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-04-27T23:22:30,708][INFO ][o.e.i.g.LocalDatabases   ] [node-1] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2022-04-27T23:22:30,709][INFO ][o.e.i.g.DatabaseRegistry ] [node-1] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-13203387310490304826/geoip-databases/AeEoJJAkSGKOyl-Wolnqjw]
[2022-04-27T23:22:32,083][INFO ][o.e.t.NettyAllocator     ] [node-1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-04-27T23:22:32,212][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2022-04-27T23:22:33,111][INFO ][o.e.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-04-27T23:22:33,486][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:33 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T23:22:33,486][WARN ][stderr                   ] [node-1] INFO: Found string system property [javax.net.ssl.trustStore]: /usr/share/elasticsearch/jdk/lib/security/cacerts.bcfks
[2022-04-27T23:22:33,487][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:33 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
[2022-04-27T23:22:33,487][WARN ][stderr                   ] [node-1] INFO: Found string system property [javax.net.ssl.trustStoreType]: BCFKS
[2022-04-27T23:22:33,487][WARN ][stderr                   ] [node-1] Apr 27, 2022 11:22:33 PM org.bouncycastle.jsse.provider.PropertyUtils getSensitiveStringSystemProperty
[2022-04-27T23:22:33,488][WARN ][stderr                   ] [node-1] INFO: Found sensitive string system property [javax.net.ssl.trustStorePassword]
[2022-04-27T23:22:34,104][INFO ][o.e.n.Node               ] [node-1] initialized
[2022-04-27T23:22:34,105][INFO ][o.e.n.Node               ] [node-1] starting ...
[2022-04-27T23:22:34,157][INFO ][o.e.x.s.c.f.PersistentCache] [node-1] persistent cache index loaded
[2022-04-27T23:22:34,334][INFO ][o.e.t.TransportService   ] [node-1] publish_address {192.168.12.226:9300}, bound_addresses {[::]:9300}
[2022-04-27T23:22:35,122][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-04-27T23:22:35,124][INFO ][o.e.c.c.Coordinator      ] [node-1] cluster UUID [CEO5_pTGTpGHS5gqgFtS1Q]
[2022-04-27T23:22:35,414][INFO ][o.e.c.s.MasterService    ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{AeEoJJAkSGKOyl-Wolnqjw}{SnhcyqGeTBmqKdcUtQ0-AQ}{192.168.12.226}{192.168.12.226:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 241, version: 283, delta: master node changed {previous [], current [{node-1}{AeEoJJAkSGKOyl-Wolnqjw}{SnhcyqGeTBmqKdcUtQ0-AQ}{192.168.12.226}{192.168.12.226:9300}{cdfhilmrstw}]}
[2022-04-27T23:22:35,566][WARN ][o.e.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:39664, profile=default}], closing connection
io.netty.handler.codec.DecoderException: java.lang.NullPointerException: Cannot invoke "org.bouncycastle.tls.TlsPeer.notifyAlertRaised(short, short, String, java.lang.Throwable)" because the return value of "org.bouncycastle.tls.TlsProtocol.getPeer()" is null
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.NullPointerException: Cannot invoke "org.bouncycastle.tls.TlsPeer.notifyAlertRaised(short, short, String, java.lang.Throwable)" because the return value of "org.bouncycastle.tls.TlsProtocol.getPeer()" is null
	at org.bouncycastle.tls.TlsProtocol.raiseAlertWarning(TlsProtocol.java:1598) ~[bctls-fips-1.0.13.jar:1.0.13]
	at org.bouncycastle.tls.TlsProtocol.handleClose(TlsProtocol.java:295) ~[bctls-fips-1.0.13.jar:1.0.13]
	at org.bouncycastle.tls.TlsProtocol.close(TlsProtocol.java:1780) ~[bctls-fips-1.0.13.jar:1.0.13]
	at org.bouncycastle.jsse.provider.ProvSSLEngine.closeOutbound(ProvSSLEngine.java:210) ~[bctls-fips-1.0.13.jar:1.0.13]
	at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1863) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1275) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1242) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
	... 16 more
[2022-04-27T23:22:35,590][WARN ][o.e.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9300, remoteAddress=/[0:0:0:0:0:0:0:1]:57960, profile=default}], closing connection
io.netty.handler.codec.DecoderException: java.lang.NullPointerException: Cannot invoke "org.bouncycastle.tls.TlsPeer.notifyAlertRaised(short, short, String, java.lang.Throwable)" because the return value of "org.bouncycastle.tls.TlsProtocol.getPeer()" is null

I do really appreciate all the help you have given so far. I understand this is not easy and is quite complex. I feel I am much closer but still missing something.

This is the elasticsearch keystore. By default an empty string is used for key derivation and this is not allowed in fips mode so you need to set a password that is large enough ( and provide it every time elastisearch starts )

I haven’t tried with bctls-fips 1.0.13 but I’d try going through the bctls user guide to see what needs to be configured. Maybe try with a different version too , I think 1.0.9 works fine IORC

I appreciate all the help, I have been able to get Elasticsearch UP and running with FIPS finally. I will post soon all the steps so others can know how to do it.

Question, in regard to Kibana, I see there was old posts about getting Kibana running on NodeJS v8 (which is now end of life as of 2019). It appears from what I can find out right now there isn't a nodeJS version that supports FIPS at this point in time. Can you speak at all on this? I'm wondering if it will not be possible to get Kibana in FIPS mode?

1 Like

I get my info from the same source (FIPS · Issue #18662 · elastic/kibana · GitHub) so I think it’s still a no AFAIU.

Kibana engineers frequent these forums too so they will keep me honest I’m sure

One other question, I am trying to get my ES ready to have a 2nd authorization for AD, when i try to have it still use the builtin default auth and i put in this code, my auth stops working, but if i remove the stanza it works again, am i missing something?

# AD Integration
xpack:
  security:
    authc:
      password_hashing.algorithm: pbkdf2_stretch
      realms:
        native:
          native1:
            order: 0

This code doesn't work, if i just remove it, my auth works again?? I just wanted to get it tested like this so when i add the 2nd realm i'm golden?

=== UPDATE

I figured out it appears I am using FILE authentication, so changing it to file fixed the issue.

I'm glad you figured it out.

In Elasticsearch 8 we've change the way this works because it caused this sort of confusion.
In ES8 the native and file realms are always included in your realm chain, unless you explicitly disable them.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.