Elasticsearch 8.15.1 in FIPS mode

Heya,

I am trying to configure a 3-node cluster in FIPS mode but TLS doesn't seem to work. Here are the system details:

  • Ubuntu 20.04
  • Elasticsearch 8.15.1
  • FIPS enabled: bc-fips-2.0.0.jar, bctls-fips-2.0.19.jar, bcpkix-fips-2.0.7 and bcutil-fips-2.0.3.jar
  • JavaTemurin17

Elasticsearch configuration:

  • elasticsearch.yml

cluster.name: fips-es-es
node.name: fips-es-1.example.com

node.roles: [ master, data ]
cluster.initial_master_nodes:
    - fips-es-1.example.com
    - fips-es-2.example.com
    - fips-es-3.example.com

path.data: /storage/elasticsearch
path.logs: /var/log/elasticsearch

network.host: fips-es-1.example.com

discovery.seed_hosts:
    - fips-es-1.example.com
    - fips-es-2.example.com
    - fips-es-3.example.com

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.cipher_suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
xpack.security.transport.ssl.supported_protocols: ["TLSv1.2", "TLSv1.3"]
#xpack.security.transport.ssl.key: "/etc/elasticsearch/certs/example.key"
#xpack.security.transport.ssl.certificate: "/etc/elasticsearch/certs/cert-and-ca.pem"
#xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/certs/CABundle.pem"]
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/example.bcfks
xpack.security.transport.ssl.keystore.password: *****
xpack.security.transport.ssl.keystore.type: "BCFKS"
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/truststoreexample.bcfks
xpack.security.transport.ssl.truststore.password: *****
xpack.security.transport.ssl.truststore.type: "BCFKS"

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.cipher_suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
xpack.security.http.ssl.supported_protocols: ["TLSv1.2", "TLSv1.3"]
#xpack.security.http.ssl.key: "/etc/elasticsearch/certs/example.key"
#xpack.security.http.ssl.certificate: "/etc/elasticsearch/certs/cert-and-ca.pem"
#xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/certs/CABundle.pem"]
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/example.bcfks
xpack.security.http.ssl.keystore.password: *****
xpack.security.http.ssl.keystore.type: "BCFKS"
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/truststoreexample.bcfks
xpack.security.http.ssl.truststore.password: *****
xpack.security.http.ssl.truststore.type: "BCFKS"

xpack.security.fips_mode.enabled: true
xpack.security.autoconfiguration.enabled: false
xpack.security.fips_mode.required_providers: ["BCFIPS", "BCJSSE"]
xpack.security.authc.password_hashing.algorithm: "pbkdf2_stretch"
  • java.security
security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
security.provider.3=SUN
security.provider.4=SunRsaSign
security.provider.5=SunEC
security.provider.6=SunJCE
security.provider.7=SunJGSS
security.provider.8=SunSASL
security.provider.9=XMLDSig
security.provider.10=SunPCSC
security.provider.11=JdkLDAP
security.provider.12=JdkSASL
security.provider.13=SunPKCS11

securerandom.strongAlgorithms=NativePRNGNonBlocking:SUN,DRBG:SUN
securerandom.source=file:/dev/urandom

ssl.KeyManagerFactory.algorithm=PKIX
ssl.TrustManagerFactory.algorithm=PKIX
  • java.policy (an official FIPS document and some community suggestions made this Frankenstein)
grant {
    // allows anyone to listen on dynamic ports
    permission java.net.SocketPermission "localhost:0", "listen";

    // "standard" properies that can be read by anyone
    permission java.util.PropertyPermission "java.version", "read";
    permission java.util.PropertyPermission "java.vendor", "read";
    permission java.util.PropertyPermission "java.vendor.url", "read";
    permission java.util.PropertyPermission "java.class.version", "read";
    permission java.util.PropertyPermission "os.name", "read";
    permission java.util.PropertyPermission "os.version", "read";
    permission java.util.PropertyPermission "os.arch", "read";
    permission java.util.PropertyPermission "file.separator", "read";
    permission java.util.PropertyPermission "path.separator", "read";
    permission java.util.PropertyPermission "line.separator", "read";
    permission java.util.PropertyPermission
                   "java.specification.version", "read";
    permission java.util.PropertyPermission "java.specification.vendor", "read";
    permission java.util.PropertyPermission "java.specification.name", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.version", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.vendor", "read";
    permission java.util.PropertyPermission
                   "java.vm.specification.name", "read";
    permission java.util.PropertyPermission "java.vm.version", "read";
    permission java.util.PropertyPermission "java.vm.vendor", "read";
    permission java.util.PropertyPermission "java.vm.name", "read";
    permission java.lang.RuntimePermission
		   "accessClassInPackage.sun.security.internal.spec";
    permission java.io.FilePermission "/usr/lib/jvm/temurin-17-jdk-amd64/lib/security/jssecacerts", "read";
    permission java.io.FilePermission "/usr/lib/jvm/temurin-17-jdk-amd64/lib/security/cacerts", "read";
    permission java.security.SecurityPermission "getProperty.jdk.tls.disabledAlgorithms";
    permission java.security.SecurityPermission "getProperty.jdk.certpath.disabledAlgorithms";
    permission java.security.SecurityPermission "getProperty.keystore.type.compat";
    permission java.io.FilePermission "/ext/tmp/elasticsearch/*", "read,write";
    permission java.io.FilePermission "/ext/tmp/elasticsearch/", "read,write";

    // https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3514.pdf
    permission java.lang.RuntimePermission "accessDeclaredMembers";
    permission org.bouncycastle.crypto.CryptoServicesPermission "tlsAlgorithmsEnabled";
    permission java.security.SecurityPermission "putProviderProperty.BCFIPS";
    permission java.lang.RuntimePermission "getProtectionDomain";
    permission java.util.PropertyPermission "java.runtime.name", "read";
    permission java.lang.RuntimePermission "accessDeclaredMembers";
    permission org.bouncycastle.crypto.CryptoServicesPermission "tlsPKCS15KeyWrapEnabled";
    Permission org.bouncycastle.crypto.CryptoServicesPermission "exportKeys";
};

// rely on the caller's socket permissions, the JSSE TLS implementation here is always allowed to connect
grant codeBase "file:$/usr/share/elasticsearch/lib/bctls-fips-2.0.19.jar" {
  permission java.net.SocketPermission "*", "connect";
};

grant codeBase "file:/usr/share/elasticsearch/lib/bc-fips.2.0.0.jar" {
    permission java.security.SecurityPermission "putProviderProperty.BCFIPS";
    permission java.security.SecurityPermission "insertProvider.BCFIPS";
    permission java.security.SecurityPermission "getProperty.BCFIPS";
    permission java.security.SecurityPermission "removeProvider.BCFIPS";
    permission java.security.SecurityPermission "clearProviderProperties.BCFIPS";
    permission java.security.SecurityPermission "getProviderProperty.BCFIPS";
};

grant codeBase "file:/usr/share/elasticsearch/lib/bctls-fips-2.0.19.jar" {
    permission java.security.SecurityPermission "putProviderProperty.BCTLS";
    permission java.security.SecurityPermission "insertProvider.BCTLS";
    permission java.security.SecurityPermission "getProperty.BCTLS";
    permission java.security.SecurityPermission "removeProvider.BCTLS";
    permission java.security.SecurityPermission "clearProviderProperties.BCTLS";
    permission java.security.SecurityPermission "getProviderProperty.BCTLS";
};

grant {
    permission java.security.SecurityPermission "getProperty.org.bouncycastle.rsa.max_size";
};
  • /etc/default/elasticsearch
ES_TMPDIR=/ext/tmp/elasticsearch
ES_JAVA_HOME=/usr/lib/jvm/temurin-17-jdk-amd64
ES_PATH_CONF=/etc/elasticsearch
ES_JAVA_OPTS="-Djava.io.tmpdir=/ext/tmp/elasticsearch -Djava.security.policy=/etc/elasticsearch/java.policy -Djava.security.properties=/etc/elasticsearch/java.security -Dorg.bouncycastle.fips.approved_only=true -Djavax.net.debug=all"
ES_KEYSTORE_PASSPHRASE_FILE="/etc/elasticsearch/kspass"
ES_STARTUP_SLEEP_TIME=5

From the elasticsearch.yml file, you can see that we have tried with both PEM and BCFKS format, but in both cases, we got the following error:

 exception caught on transport layer [Netty4TcpChannel{localAddress=/10.17.11.202:9300, remoteAddress=/10.23.81.29:44258, profile=default}], closing connection
io.netty.handler.codec.DecoderException: java.lang.NullPointerException: Cannot invoke "org.bouncycastle.tls.TlsPeer.notifyAlertRaised(short, short, String, java.lang.Throwable)" because the return value of "org.bouncycastle.tls.TlsProtocol.getPeer()" is null

Connection from one node to another over TLS doesn't work, it fails on the TLS handshake:

CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 339 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

There is a statement in the official documentation

Elasticsearch has been tested with Bouncy Castle’s [bc-fips 1.0.2.4] and [bctls-fips 1.0.17]

but unfortunately, even with those versions, we got exactly the same error as above.

Please help guys, I am in the dark :exploding_head: Let me know if I have to provide something more.

My guess is that this configuration is incorrect.
What's in those bcfks files?

Hey Tim, thanks for the reply.

It's a BCFKS format created from the JKS keystore with the following command:

keytool -importkeystore -srckeystore exmp.jks -srcstoretype JKS -srcstorepass randompass -destkeystore exmp.bcfks -deststorepass randompass -deststoretype BCFKS -providername BCFIPS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /usr/share/elasticsearch/lib/bc-fips-2.0.0.jar -J-Djava.security.properties=/etc/elasticsearch/java.security

It doesn't work with PEM format either, it has the same error...

Can you DISABLE FIPS mode and try again ? In our case FIPS has caused a lot of issues

In case if your security team does allow you to disable FIPS below is how we disabled it on RHEL 8
fips-mode-setup --check
fips-mode-setup --disable
once done REBOOT server

No, we can't disable it, it's a security and compliance requirement.

I'd also like to add that cluster works ok in FIPS mode with the following setup:

  • bc-fips-1.0.2.jar, bctls-fips-1.0.11.jar, bcpkix-fips-1.0.5.jar
  • Elasticsearch 7.16+
  • Java Temurin 11
  • both PEM and BCFKS format

But these bc libs are obsoleted, BC-FJA 2.0.0 was released few months ago (30 July, 2024).

Sorry, I missed that you were running BC 2.0

We do not currently support BC-FIPS 2.0

1 Like

Thanks for the valuable information. Do you have any clues about when we can expect it?

I saw in the documentation that it has been tested with some newer versions

Elasticsearch has been tested with Bouncy Castle’s [bc-fips 1.0.2.4] and [bctls-fips 1.0.17]

but interestingly enough, it doesn't work with these versions either.

From your earlier message, it sounded like you had it working.
If you have a problem with an officially supported version of BC-FIPS we can help debug that.

Can you describe the problem and post any error messages you get with 1.0.2.4 ?

Summary:

So, it does work with the following bc libs

  • bc-fips-1.0.2.jar
  • bctls-fips-1.0.11.jar
  • bcpkix-fips-1.0.5.jar

with JavaTemurin11 and Elastic v7, but these libs are outdated and should be updated.

We wanted to update the Java version as well, so the setup was JavaTemurin17, Elasticsearch v8.15.1, and the latest BC 2.0 because bc-fips-1.0.2.4.jar has also been moved to the historical list due to sunsetting in August this year. You can check Certificate #4616.

So we tested two options:

  1. Java17, Elastic 8.15.1, bc-fips-2.0.0.jar, bctls-fips-2.0.19.jar, bcpkix-fips-2.0.7, bcutil-fips-2.0.3.jar

  2. Java17, Elastic 8.15.1, bc-fips-1.0.2.4.jar, bctls-fips-1.0.17, bcpkix-fips-1.0.7.jar

and both options were throwing the same error:

exception caught on transport layer [Netty4TcpChannel{localAddress=/10.17.11.202:9300, remoteAddress=/10.23.81.29:44258, profile=default}], closing connection
io.netty.handler.codec.DecoderException: java.lang.NullPointerException: Cannot invoke "org.bouncycastle.tls.TlsPeer.notifyAlertRaised(short, short, String, java.lang.Throwable)" because the return value of "org.bouncycastle.tls.TlsProtocol.getPeer()" is null

Cert #4743 is a new certificate for BC 2.0.

hey @TimV,

in case you missed my question in the earlier comment ^

When can we expect support for BC-FIPS 2.0?