Elasticsearch wont start with FIPS Mode Enabled

Zoheb_Khan · August 9, 2021, 4:42am

I am not able to bring-up elasticsearch service in FIPS mode 140-2. I keep getting exception :
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.common.ssl.SslConfigException: failed to find a X509ExtendedTrustManager in the trust manager factory for [PKIX] and truststore [null]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.10.2.jar:7.10.2]

Really appreciate if someone can look into this.

I have following configuration.
OS:
"name" : "Linux",
"pretty_name" : "CentOS Linux 7 (Core)",
"arch" : "amd64",
"version" : "4.9.184-35.el7.centos.x86_64",

JVM :
"version" : "1.8.0_282",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "25.282-b08",
"vm_vendor" : "Red Hat, Inc.",

BC FIPS provider :
bc-fips-1.0.2.jar
bcpg-fips-1.0.5.jar
bctls-fips-1.0.10.jar

ES :
7.10.2 with X-pack configured (YML config below)
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.algorithm: PKIX
xpack.security.http.ssl.truststore.algorithm: PKIX
xpack.security.http.ssl.keystore.type: BCFKS
xpack.security.http.ssl.truststore.type: BCFKS
xpack.security.http.ssl.keystore.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64/jre/bin/elasticsearch.keystore.bcfks
xpack.security.http.ssl.truststore.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64/jre/bin/elasticsearch.truststore.bcfks
xpack.security.http.ssl.client_authentication: optional
xpack.security.fips_mode.enabled: true
xpack.security.http.ssl.supported_protocols: TLSv1, TLSv1.1, TLSv1.2, TLSv1.3
xpack.security.authc.password_hashing.algorithm: pbkdf2

JAVA Security file :

security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
security.provider.4=sun.security.ec.SunEC
security.provider.5=com.sun.net.ssl.internal.ssl.Provider BCFIPS
security.provider.6=com.sun.crypto.provider.SunJCE
security.provider.7=sun.security.jgss.SunProvider
security.provider.8=com.sun.security.sasl.Provider
security.provider.9=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.10=sun.security.smartcardio.SunPCSC
security.provider.11=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:DEFRND[HmacSHA512];ENABLE{ALL};
security.provider.12=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
crypto.policy=unlimited
securerandom.source=file:/dev/random
securerandom.strongAlgorithms=NativePRNGBlocking:SUN
login.configuration.provider=sun.security.provider.ConfigFile
policy.provider=sun.security.provider.PolicyFile
policy.url.1=file:${java.home}/lib/security/java.policy
policy.url.2=file:${user.home}/.java.policy
policy.expandProperties=true
policy.allowSystemProperty=true
policy.ignoreIdentityScope=false
keystore.type=BCFKS
keystore.type.compat=true
package.access=sun.,
com.sun.xml.internal.,
com.sun.imageio.,
com.sun.istack.internal.,
com.sun.jmx.,
com.sun.media.sound.,
com.sun.naming.internal.,
com.sun.proxy.,
com.sun.corba.se.,
com.sun.org.apache.bcel.internal.,
com.sun.org.apache.regexp.internal.,
com.sun.org.apache.xerces.internal.,
com.sun.org.apache.xpath.internal.,
com.sun.org.apache.xalan.internal.extensions.,
com.sun.org.apache.xalan.internal.lib.,
com.sun.org.apache.xalan.internal.res.,
com.sun.org.apache.xalan.internal.templates.,
com.sun.org.apache.xalan.internal.utils.,
com.sun.org.apache.xalan.internal.xslt.,
com.sun.org.apache.xalan.internal.xsltc.cmdline.,
com.sun.org.apache.xalan.internal.xsltc.compiler.,
com.sun.org.apache.xalan.internal.xsltc.trax.,
com.sun.org.apache.xalan.internal.xsltc.util.,
com.sun.org.apache.xml.internal.res.,
com.sun.org.apache.xml.internal.resolver.helpers.,
com.sun.org.apache.xml.internal.resolver.readers.,
com.sun.org.apache.xml.internal.security.,
com.sun.org.apache.xml.internal.serializer.utils.,
com.sun.org.apache.xml.internal.utils.,
com.sun.org.glassfish.,
com.oracle.xmlns.internal.,
com.oracle.webservices.internal.,
oracle.jrockit.jfr.,
org.jcp.xml.dsig.internal.,
jdk.internal.,
jdk.nashorn.internal.,
jdk.nashorn.tools.,
com.sun.activation.registries.,
org.GNOME.Accessibility.,
org.GNOME.Bonobo.

package.definition=sun.,
com.sun.xml.internal.,
com.sun.imageio.,
com.sun.istack.internal.,
com.sun.jmx.,
com.sun.media.sound.,
com.sun.naming.internal.,
com.sun.proxy.,
com.sun.corba.se.,
com.sun.org.apache.bcel.internal.,
com.sun.org.apache.regexp.internal.,
com.sun.org.apache.xerces.internal.,
com.sun.org.apache.xpath.internal.,
com.sun.org.apache.xalan.internal.extensions.,
com.sun.org.apache.xalan.internal.lib.,
com.sun.org.apache.xalan.internal.res.,
com.sun.org.apache.xalan.internal.templates.,
com.sun.org.apache.xalan.internal.utils.,
com.sun.org.apache.xalan.internal.xslt.,
com.sun.org.apache.xalan.internal.xsltc.cmdline.,
com.sun.org.apache.xalan.internal.xsltc.compiler.,
com.sun.org.apache.xalan.internal.xsltc.trax.,
com.sun.org.apache.xalan.internal.xsltc.util.,
com.sun.org.apache.xml.internal.res.,
com.sun.org.apache.xml.internal.resolver.helpers.,
com.sun.org.apache.xml.internal.resolver.readers.,
com.sun.org.apache.xml.internal.security.,
com.sun.org.apache.xml.internal.serializer.utils.,
com.sun.org.apache.xml.internal.utils.,
com.sun.org.glassfish.,
com.oracle.xmlns.internal.,
com.oracle.webservices.internal.,
oracle.jrockit.jfr.,
org.jcp.xml.dsig.internal.,
jdk.internal.,
jdk.nashorn.internal.,
jdk.nashorn.tools.,
com.sun.activation.registries.,
org.GNOME.Accessibility.,
org.GNOME.Bonobo.
security.overridePropertiesFile=true
ssl.KeyManagerFactory.algorithm=SunX509
ssl.TrustManagerFactory.algorithm=PKIX
networkaddress.cache.negative.ttl=10
krb5.kdc.bad.policy = tryLast
jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer,
RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768,
EC keySize < 224
jdk.tls.legacyAlgorithms=
K_NULL, C_NULL, M_NULL,
DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_anon_EXPORT, DH_DSS_EXPORT,
DH_RSA_EXPORT, RSA_EXPORT,
DH_anon, ECDH_anon,
RC4_128, RC4_40, DES_CBC, DES40_CBC,
3DES_EDE_CBC
jdk.xml.dsig.secureValidationPolicy=
disallowAlg XSL Transformations (XSLT),
maxTransforms 5,
maxReferences 30,
disallowReferenceUriSchemes file http https,
minKeySize RSA 1024,
minKeySize DSA 1024,
noDuplicateIds,
noRetrievalMethodLoops

Thank you!
Zoheb

ikakavas · August 9, 2021, 5:14am

WARNING We are not FIPS 140-2 experts and I doubt you will find one in these forums. Please don't use the information exchanged here as a proof of FIPS 140-2 compliance and reach out to accredited and experienced 3rd parties for your production systems.

This is mostly a "how to configure my JRE" problem and not an Elasticsearch one. There are a number of things that are wrong in your JRE confiuration

Please use 7.11, we resolved a number of bugs ( not something to do with what you describe above, but still ) related to FIPS 140 support.
```
  xpack.security.http.ssl.keystore.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64/jre/bin/elasticsearch.keystore.bcfks
  xpack.security.http.ssl.truststore.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64/jre/bin/elasticsearch.truststore.bcfks
```
How did you create this elasticsearch.keystore.bcfks and elasticsearch.truststore.bcfks files? The naming alludes to elasticsearch.keystore which is an internal secure settings store for elasticsearch and can't be used as a TLS keystore. Additionally, elasticsearch (via the Java Security Manager) won't allow you to read files outside elasticsearch's configuration directory, so you need to place all your configuration file somewhere in there.

In your java security file, the list of providers is wrong, you need to remove the first line, and move

security.provider.11=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:DEFRND[HmacSHA512];ENABLE{ALL};
security.provider.12=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS crypto.policy=unlimited

to be 1 and 2 respectively,

You need to convert your JVM cacerts keystore to BCFKS and password protect it and set javax.net.ssl.trustStorePassword and javax.net.ssl.keyStorePassword as system properties either in jvm.properties or when running elasticsearch
How do you make the BCFIPS provider JARs available to your system ? In Java 8 you can still use /lib/ext in your JRE and put them there

This is probably an incomplete list of things to change/adjust, hope it helps

Zoheb_Khan · August 9, 2021, 2:44pm

Thank you for a prompt response. Let me review and get back if I have any follow-up questions.

system · September 6, 2021, 2:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ElasticSearch FIPS (BouncyCastle) Elasticsearch elastic-stack-security	12	2854	June 2, 2022
Issues trying to enable FIPS 140-2 on Centos 8 Elasticsearch elastic-stack-security	3	1279	April 21, 2022
Elasticsearch 7.6.2 fails to start up with BCFKS enforcement Elasticsearch	2	504	March 26, 2021
Exception while enabling fips in es7.17 Elasticsearch	8	170	April 18, 2024
Elastic wont start - java.io.IOException: keystore password was incorrect Elasticsearch elastic-stack-security	5	8341	November 21, 2019

Elasticsearch wont start with FIPS Mode Enabled

JAVA Security file :

Related topics