Elasticsearch 8.15.1 in FIPS mode

fox9 · September 23, 2024, 4:25pm

Heya,

I am trying to configure a 3-node cluster in FIPS mode but TLS doesn't seem to work. Here are the system details:

Ubuntu 20.04
Elasticsearch 8.15.1
FIPS enabled: bc-fips-2.0.0.jar, bctls-fips-2.0.19.jar, bcpkix-fips-2.0.7 and bcutil-fips-2.0.3.jar
JavaTemurin17

Elasticsearch configuration:

elasticsearch.yml


cluster.name: fips-es-es
node.name: fips-es-1.example.com

node.roles: [ master, data ]
cluster.initial_master_nodes:
    - fips-es-1.example.com
    - fips-es-2.example.com
    - fips-es-3.example.com

path.data: /storage/elasticsearch
path.logs: /var/log/elasticsearch

network.host: fips-es-1.example.com

discovery.seed_hosts:
    - fips-es-1.example.com
    - fips-es-2.example.com
    - fips-es-3.example.com

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.cipher_suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
xpack.security.transport.ssl.supported_protocols: ["TLSv1.2", "TLSv1.3"]
#xpack.security.transport.ssl.key: "/etc/elasticsearch/certs/example.key"
#xpack.security.transport.ssl.certificate: "/etc/elasticsearch/certs/cert-and-ca.pem"
#xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/certs/CABundle.pem"]
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/example.bcfks
xpack.security.transport.ssl.keystore.password: *****
xpack.security.transport.ssl.keystore.type: "BCFKS"
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/truststoreexample.bcfks
xpack.security.transport.ssl.truststore.password: *****
xpack.security.transport.ssl.truststore.type: "BCFKS"

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.cipher_suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
xpack.security.http.ssl.supported_protocols: ["TLSv1.2", "TLSv1.3"]
#xpack.security.http.ssl.key: "/etc/elasticsearch/certs/example.key"
#xpack.security.http.ssl.certificate: "/etc/elasticsearch/certs/cert-and-ca.pem"
#xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/certs/CABundle.pem"]
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/example.bcfks
xpack.security.http.ssl.keystore.password: *****
xpack.security.http.ssl.keystore.type: "BCFKS"
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/truststoreexample.bcfks
xpack.security.http.ssl.truststore.password: *****
xpack.security.http.ssl.truststore.type: "BCFKS"

xpack.security.fips_mode.enabled: true
xpack.security.autoconfiguration.enabled: false
xpack.security.fips_mode.required_providers: ["BCFIPS", "BCJSSE"]
xpack.security.authc.password_hashing.algorithm: "pbkdf2_stretch"

java.security

security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
security.provider.3=SUN
security.provider.4=SunRsaSign
security.provider.5=SunEC
security.provider.6=SunJCE
security.provider.7=SunJGSS
security.provider.8=SunSASL
security.provider.9=XMLDSig
security.provider.10=SunPCSC
security.provider.11=JdkLDAP
security.provider.12=JdkSASL
security.provider.13=SunPKCS11

securerandom.strongAlgorithms=NativePRNGNonBlocking:SUN,DRBG:SUN
securerandom.source=file:/dev/urandom

ssl.KeyManagerFactory.algorithm=PKIX
ssl.TrustManagerFactory.algorithm=PKIX

java.policy (an official FIPS document and some community suggestions made this Frankenstein)

grant {
    // allows anyone to listen on dynamic ports
    permission java.net.SocketPermission "localhost:0", "listen";

    // "standard" properies that can be read by anyone
    permission java.util.PropertyPermission "java.version", "read";
    permission java.util.PropertyPermission "java.vendor", "read";
    permission java.util.PropertyPermission "java.vendor.url", "read";
    permission java.util.PropertyPermission "java.class.version", "read";
    permission java.util.PropertyPermission "os.name", "read";
    permission java.util.PropertyPermission "os.version", "read";
    permission java.util.PropertyPermission "os.arch", "read";
    permission java.util.PropertyPermission "file.separator", "read";
    permission java.util.PropertyPermission "path.separator", "read";
    permission java.util.PropertyPermission "line.separator", "read";
    permission java.util.PropertyPermission
                   "java.specification.version", "read";
    permission java.util.PropertyPermission "java.specification.vendor", "read";
    permission java.util.PropertyPermission "java.specification.name", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.version", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.vendor", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.name", "read";
    permission java.util.PropertyPermission "java.vm.version", "read";
    permission java.util.PropertyPermission "java.vm.vendor", "read";
    permission java.util.PropertyPermission "java.vm.name", "read";
    permission java.lang.RuntimePermission
		   "accessClassInPackage.sun.security.internal.spec";
    permission java.io.FilePermission "/usr/lib/jvm/temurin-17-jdk-amd64/lib/security/jssecacerts", "read";
    permission java.io.FilePermission "/usr/lib/jvm/temurin-17-jdk-amd64/lib/security/cacerts", "read";
    permission java.security.SecurityPermission "getProperty.jdk.tls.disabledAlgorithms";
    permission java.security.SecurityPermission "getProperty.jdk.certpath.disabledAlgorithms";
    permission java.security.SecurityPermission "getProperty.keystore.type.compat";
    permission java.io.FilePermission "/ext/tmp/elasticsearch/*", "read,write";
    permission java.io.FilePermission "/ext/tmp/elasticsearch/", "read,write";

    // https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3514.pdf
    permission java.lang.RuntimePermission "accessDeclaredMembers";
    permission org.bouncycastle.crypto.CryptoServicesPermission "tlsAlgorithmsEnabled";
    permission java.security.SecurityPermission "putProviderProperty.BCFIPS";
    permission java.lang.RuntimePermission "getProtectionDomain";
    permission java.util.PropertyPermission "java.runtime.name", "read";
    permission java.lang.RuntimePermission "accessDeclaredMembers";
    permission org.bouncycastle.crypto.CryptoServicesPermission "tlsPKCS15KeyWrapEnabled";
    Permission org.bouncycastle.crypto.CryptoServicesPermission "exportKeys";
};

// rely on the caller's socket permissions, the JSSE TLS implementation here is always allowed to connect
grant codeBase "file:$/usr/share/elasticsearch/lib/bctls-fips-2.0.19.jar" {
  permission java.net.SocketPermission "*", "connect";
};

grant codeBase "file:/usr/share/elasticsearch/lib/bc-fips.2.0.0.jar" {
    permission java.security.SecurityPermission "putProviderProperty.BCFIPS";
    permission java.security.SecurityPermission "insertProvider.BCFIPS";
    permission java.security.SecurityPermission "getProperty.BCFIPS";
    permission java.security.SecurityPermission "removeProvider.BCFIPS";
    permission java.security.SecurityPermission "clearProviderProperties.BCFIPS";
    permission java.security.SecurityPermission "getProviderProperty.BCFIPS";
};

grant codeBase "file:/usr/share/elasticsearch/lib/bctls-fips-2.0.19.jar" {
    permission java.security.SecurityPermission "putProviderProperty.BCTLS";
    permission java.security.SecurityPermission "insertProvider.BCTLS";
    permission java.security.SecurityPermission "getProperty.BCTLS";
    permission java.security.SecurityPermission "removeProvider.BCTLS";
    permission java.security.SecurityPermission "clearProviderProperties.BCTLS";
    permission java.security.SecurityPermission "getProviderProperty.BCTLS";
};

grant {
    permission java.security.SecurityPermission "getProperty.org.bouncycastle.rsa.max_size";
};

/etc/default/elasticsearch

ES_TMPDIR=/ext/tmp/elasticsearch
ES_JAVA_HOME=/usr/lib/jvm/temurin-17-jdk-amd64
ES_PATH_CONF=/etc/elasticsearch
ES_JAVA_OPTS="-Djava.io.tmpdir=/ext/tmp/elasticsearch -Djava.security.policy=/etc/elasticsearch/java.policy -Djava.security.properties=/etc/elasticsearch/java.security -Dorg.bouncycastle.fips.approved_only=true -Djavax.net.debug=all"
ES_KEYSTORE_PASSPHRASE_FILE="/etc/elasticsearch/kspass"
ES_STARTUP_SLEEP_TIME=5

From the elasticsearch.yml file, you can see that we have tried with both PEM and BCFKS format, but in both cases, we got the following error:

 exception caught on transport layer [Netty4TcpChannel{localAddress=/10.17.11.202:9300, remoteAddress=/10.23.81.29:44258, profile=default}], closing connection
io.netty.handler.codec.DecoderException: java.lang.NullPointerException: Cannot invoke "org.bouncycastle.tls.TlsPeer.notifyAlertRaised(short, short, String, java.lang.Throwable)" because the return value of "org.bouncycastle.tls.TlsProtocol.getPeer()" is null

Connection from one node to another over TLS doesn't work, it fails on the TLS handshake:

CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 339 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

There is a statement in the official documentation

Elasticsearch has been tested with Bouncy Castle’s [bc-fips 1.0.2.4] and [bctls-fips 1.0.17]

but unfortunately, even with those versions, we got exactly the same error as above.

Please help guys, I am in the dark Let me know if I have to provide something more.

TimV · September 24, 2024, 12:33am

fox9:

xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/example.bcfks
xpack.security.transport.ssl.keystore.password: *****
xpack.security.transport.ssl.keystore.type: "BCFKS"
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/truststoreexample.bcfks
xpack.security.transport.ssl.truststore.password: *****
xpack.security.transport.ssl.truststore.type: "BCFKS"

My guess is that this configuration is incorrect.
What's in those bcfks files?

fox9 · September 24, 2024, 6:47am

Hey Tim, thanks for the reply.

It's a BCFKS format created from the JKS keystore with the following command:

keytool -importkeystore -srckeystore exmp.jks -srcstoretype JKS -srcstorepass randompass -destkeystore exmp.bcfks -deststorepass randompass -deststoretype BCFKS -providername BCFIPS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /usr/share/elasticsearch/lib/bc-fips-2.0.0.jar -J-Djava.security.properties=/etc/elasticsearch/java.security

It doesn't work with PEM format either, it has the same error...

devin · September 24, 2024, 3:08pm

Can you DISABLE FIPS mode and try again ? In our case FIPS has caused a lot of issues

devin · September 24, 2024, 3:13pm

In case if your security team does allow you to disable FIPS below is how we disabled it on RHEL 8
fips-mode-setup --check
fips-mode-setup --disable
once done REBOOT server

fox9 · September 24, 2024, 4:42pm

No, we can't disable it, it's a security and compliance requirement.

fox9 · September 24, 2024, 7:59pm

I'd also like to add that cluster works ok in FIPS mode with the following setup:

bc-fips-1.0.2.jar, bctls-fips-1.0.11.jar, bcpkix-fips-1.0.5.jar
Elasticsearch 7.16+
Java Temurin 11
both PEM and BCFKS format

But these bc libs are obsoleted, BC-FJA 2.0.0 was released few months ago (30 July, 2024).

TimV · September 25, 2024, 2:56am

Sorry, I missed that you were running BC 2.0

We do not currently support BC-FIPS 2.0

fox9 · September 25, 2024, 6:58am

Thanks for the valuable information. Do you have any clues about when we can expect it?

I saw in the documentation that it has been tested with some newer versions

Elasticsearch has been tested with Bouncy Castle’s [bc-fips 1.0.2.4] and [bctls-fips 1.0.17]

but interestingly enough, it doesn't work with these versions either.

TimV · September 25, 2024, 11:25pm

From your earlier message, it sounded like you had it working.
If you have a problem with an officially supported version of BC-FIPS we can help debug that.

Can you describe the problem and post any error messages you get with 1.0.2.4 ?

fox9 · September 26, 2024, 7:19am

Summary:

So, it does work with the following bc libs

bc-fips-1.0.2.jar
bctls-fips-1.0.11.jar
bcpkix-fips-1.0.5.jar

with JavaTemurin11 and Elastic v7, but these libs are outdated and should be updated.

We wanted to update the Java version as well, so the setup was JavaTemurin17, Elasticsearch v8.15.1, and the latest BC 2.0 because bc-fips-1.0.2.4.jar has also been moved to the historical list due to sunsetting in August this year. You can check Certificate #4616.

So we tested two options:

Java17, Elastic 8.15.1, bc-fips-2.0.0.jar, bctls-fips-2.0.19.jar, bcpkix-fips-2.0.7, bcutil-fips-2.0.3.jar
Java17, Elastic 8.15.1, bc-fips-1.0.2.4.jar, bctls-fips-1.0.17, bcpkix-fips-1.0.7.jar

and both options were throwing the same error:

exception caught on transport layer [Netty4TcpChannel{localAddress=/10.17.11.202:9300, remoteAddress=/10.23.81.29:44258, profile=default}], closing connection
io.netty.handler.codec.DecoderException: java.lang.NullPointerException: Cannot invoke "org.bouncycastle.tls.TlsPeer.notifyAlertRaised(short, short, String, java.lang.Throwable)" because the return value of "org.bouncycastle.tls.TlsProtocol.getPeer()" is null

Cert #4743 is a new certificate for BC 2.0.

fox9 · October 4, 2024, 3:29pm

hey @TimV,

in case you missed my question in the earlier comment ^

When can we expect support for BC-FIPS 2.0?

TimV · October 11, 2024, 2:25am

I'm sorry, we don't offer timeframes for future features.
We're aware there is interest in support for BC-FIPS 2.0

Topic		Replies	Views
ElasticSearch FIPS (BouncyCastle) Elasticsearch elastic-stack-security	12	2888	June 2, 2022
TLS/SSL enabled cluster , SSL is working TSL not working with p12 cert password protected Elasticsearch elastic-stack-security	22	5547	December 30, 2018
Using TLS with PFX from third party in 7.11 Elasticsearch elastic-stack-security	2	1336	April 7, 2021
Elasticsearch wont start with FIPS Mode Enabled Elasticsearch elastic-stack-security	3	1342	September 6, 2021
TLS not working with x-pack for es cluster Elasticsearch	9	4012	December 25, 2017

Elasticsearch 8.15.1 in FIPS mode

Related topics