Elasticsearch wont start with FIPS Mode Enabled

I am not able to bring-up elasticsearch service in FIPS mode 140-2. I keep getting exception :
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.common.ssl.SslConfigException: failed to find a X509ExtendedTrustManager in the trust manager factory for [PKIX] and truststore [null]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.10.2.jar:7.10.2]

Really appreciate if someone can look into this.

I have following configuration.
OS:
"name" : "Linux",
"pretty_name" : "CentOS Linux 7 (Core)",
"arch" : "amd64",
"version" : "4.9.184-35.el7.centos.x86_64",

JVM :
"version" : "1.8.0_282",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "25.282-b08",
"vm_vendor" : "Red Hat, Inc.",

BC FIPS provider :
bc-fips-1.0.2.jar
bcpg-fips-1.0.5.jar
bctls-fips-1.0.10.jar

ES :
7.10.2 with X-pack configured (YML config below)
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.algorithm: PKIX
xpack.security.http.ssl.truststore.algorithm: PKIX
xpack.security.http.ssl.keystore.type: BCFKS
xpack.security.http.ssl.truststore.type: BCFKS
xpack.security.http.ssl.keystore.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64/jre/bin/elasticsearch.keystore.bcfks
xpack.security.http.ssl.truststore.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64/jre/bin/elasticsearch.truststore.bcfks
xpack.security.http.ssl.client_authentication: optional
xpack.security.fips_mode.enabled: true
xpack.security.http.ssl.supported_protocols: TLSv1, TLSv1.1, TLSv1.2, TLSv1.3
xpack.security.authc.password_hashing.algorithm: pbkdf2

JAVA Security file :

security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
security.provider.4=sun.security.ec.SunEC
security.provider.5=com.sun.net.ssl.internal.ssl.Provider BCFIPS
security.provider.6=com.sun.crypto.provider.SunJCE
security.provider.7=sun.security.jgss.SunProvider
security.provider.8=com.sun.security.sasl.Provider
security.provider.9=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.10=sun.security.smartcardio.SunPCSC
security.provider.11=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:DEFRND[HmacSHA512];ENABLE{ALL};
security.provider.12=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
crypto.policy=unlimited
securerandom.source=file:/dev/random
securerandom.strongAlgorithms=NativePRNGBlocking:SUN
login.configuration.provider=sun.security.provider.ConfigFile
policy.provider=sun.security.provider.PolicyFile
policy.url.1=file:${java.home}/lib/security/java.policy
policy.url.2=file:${user.home}/.java.policy
policy.expandProperties=true
policy.allowSystemProperty=true
policy.ignoreIdentityScope=false
keystore.type=BCFKS
keystore.type.compat=true
package.access=sun.,
com.sun.xml.internal.,
com.sun.imageio.,
com.sun.istack.internal.,
com.sun.jmx.,
com.sun.media.sound.,
com.sun.naming.internal.,
com.sun.proxy.,
com.sun.corba.se.,
com.sun.org.apache.bcel.internal.,
com.sun.org.apache.regexp.internal.,
com.sun.org.apache.xerces.internal.,
com.sun.org.apache.xpath.internal.,
com.sun.org.apache.xalan.internal.extensions.,
com.sun.org.apache.xalan.internal.lib.,
com.sun.org.apache.xalan.internal.res.,
com.sun.org.apache.xalan.internal.templates.,
com.sun.org.apache.xalan.internal.utils.,
com.sun.org.apache.xalan.internal.xslt.,
com.sun.org.apache.xalan.internal.xsltc.cmdline.,
com.sun.org.apache.xalan.internal.xsltc.compiler.,
com.sun.org.apache.xalan.internal.xsltc.trax.,
com.sun.org.apache.xalan.internal.xsltc.util.,
com.sun.org.apache.xml.internal.res.,
com.sun.org.apache.xml.internal.resolver.helpers.,
com.sun.org.apache.xml.internal.resolver.readers.,
com.sun.org.apache.xml.internal.security.,
com.sun.org.apache.xml.internal.serializer.utils.,
com.sun.org.apache.xml.internal.utils.,
com.sun.org.glassfish.,
com.oracle.xmlns.internal.,
com.oracle.webservices.internal.,
oracle.jrockit.jfr.,
org.jcp.xml.dsig.internal.,
jdk.internal.,
jdk.nashorn.internal.,
jdk.nashorn.tools.,
com.sun.activation.registries.,
org.GNOME.Accessibility.,
org.GNOME.Bonobo.

package.definition=sun.,
com.sun.xml.internal.,
com.sun.imageio.,
com.sun.istack.internal.,
com.sun.jmx.,
com.sun.media.sound.,
com.sun.naming.internal.,
com.sun.proxy.,
com.sun.corba.se.,
com.sun.org.apache.bcel.internal.,
com.sun.org.apache.regexp.internal.,
com.sun.org.apache.xerces.internal.,
com.sun.org.apache.xpath.internal.,
com.sun.org.apache.xalan.internal.extensions.,
com.sun.org.apache.xalan.internal.lib.,
com.sun.org.apache.xalan.internal.res.,
com.sun.org.apache.xalan.internal.templates.,
com.sun.org.apache.xalan.internal.utils.,
com.sun.org.apache.xalan.internal.xslt.,
com.sun.org.apache.xalan.internal.xsltc.cmdline.,
com.sun.org.apache.xalan.internal.xsltc.compiler.,
com.sun.org.apache.xalan.internal.xsltc.trax.,
com.sun.org.apache.xalan.internal.xsltc.util.,
com.sun.org.apache.xml.internal.res.,
com.sun.org.apache.xml.internal.resolver.helpers.,
com.sun.org.apache.xml.internal.resolver.readers.,
com.sun.org.apache.xml.internal.security.,
com.sun.org.apache.xml.internal.serializer.utils.,
com.sun.org.apache.xml.internal.utils.,
com.sun.org.glassfish.,
com.oracle.xmlns.internal.,
com.oracle.webservices.internal.,
oracle.jrockit.jfr.,
org.jcp.xml.dsig.internal.,
jdk.internal.,
jdk.nashorn.internal.,
jdk.nashorn.tools.,
com.sun.activation.registries.,
org.GNOME.Accessibility.,
org.GNOME.Bonobo.
security.overridePropertiesFile=true
ssl.KeyManagerFactory.algorithm=SunX509
ssl.TrustManagerFactory.algorithm=PKIX
networkaddress.cache.negative.ttl=10
krb5.kdc.bad.policy = tryLast
jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer,
RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768,
EC keySize < 224
jdk.tls.legacyAlgorithms=
K_NULL, C_NULL, M_NULL,
DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_anon_EXPORT, DH_DSS_EXPORT,
DH_RSA_EXPORT, RSA_EXPORT,
DH_anon, ECDH_anon,
RC4_128, RC4_40, DES_CBC, DES40_CBC,
3DES_EDE_CBC
jdk.xml.dsig.secureValidationPolicy=
disallowAlg XSL Transformations (XSLT),
maxTransforms 5,
maxReferences 30,
disallowReferenceUriSchemes file http https,
minKeySize RSA 1024,
minKeySize DSA 1024,
noDuplicateIds,
noRetrievalMethodLoops


Thank you!
Zoheb

WARNING We are not FIPS 140-2 experts and I doubt you will find one in these forums. Please don't use the information exchanged here as a proof of FIPS 140-2 compliance and reach out to accredited and experienced 3rd parties for your production systems.

This is mostly a "how to configure my JRE" problem and not an Elasticsearch one. There are a number of things that are wrong in your JRE confiuration

  1. Please use 7.11, we resolved a number of bugs ( not something to do with what you describe above, but still ) related to FIPS 140 support.

  2.   xpack.security.http.ssl.keystore.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64/jre/bin/elasticsearch.keystore.bcfks
      xpack.security.http.ssl.truststore.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64/jre/bin/elasticsearch.truststore.bcfks
    

    How did you create this elasticsearch.keystore.bcfks and elasticsearch.truststore.bcfks files? The naming alludes to elasticsearch.keystore which is an internal secure settings store for elasticsearch and can't be used as a TLS keystore. Additionally, elasticsearch (via the Java Security Manager) won't allow you to read files outside elasticsearch's configuration directory, so you need to place all your configuration file somewhere in there.

  3. In your java security file, the list of providers is wrong, you need to remove the first line, and move

    security.provider.11=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:DEFRND[HmacSHA512];ENABLE{ALL};
    security.provider.12=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS crypto.policy=unlimited
    

    to be 1 and 2 respectively,

  4. You need to convert your JVM cacerts keystore to BCFKS and password protect it and set javax.net.ssl.trustStorePassword and javax.net.ssl.keyStorePassword as system properties either in jvm.properties or when running elasticsearch

  5. How do you make the BCFIPS provider JARs available to your system ? In Java 8 you can still use /lib/ext in your JRE and put them there

This is probably an incomplete list of things to change/adjust, hope it helps

Thank you for a prompt response. Let me review and get back if I have any follow-up questions.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.