Elasticsearch is incompatible with new versions of openjdk from redhat

Redhat introduced a new change in their openjdk packages (starting from 1.8.0-345.b01-5.el8_7) for rhel 8.7 where they check for the system cacert automatically when you initialize a new trustmanager object (even without passing any cert).

This leads to a permissiondenied errors on the system cacert file in the Elasticsearch bootstrap chain even if

xpack.security.enabled: false

I even added these for good measures

xpack.security.transport.ssl.enabled: false

Error

[2022-11-09T07:52:26,851][INFO ][o.e.n.Node               ] [odrzse24-storage-node-1] JVM arguments
[-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75,
-XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60,
-Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m,
-Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true,
-XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true,
-Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0,
-Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true,
-Dlog4j2.formatMsgNoLookups=true,
-Djava.io.tmpdir=/tmp/elasticsearch-1598213987235406706,
-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch,
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails,
-XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution,
-XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log,
-XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m,
-Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch,
-Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2022-11-09T07:52:27,096][ERROR][o.e.b.Bootstrap          ] [odrzse24-storage-node-1] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:616) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.23.jar:6.8.23]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more
Caused by: java.lang.ExceptionInInitializerError
	at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56) ~[?:?]
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_352]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:279) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:48) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_352]
	at java.security.AccessController.checkPermission(AccessController.java:886) ~[?:1.8.0_352]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_352]
	at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_352]
	at java.io.File.exists(File.java:825) ~[?:1.8.0_352]
	at sun.security.tools.KeyStoreUtil.getCacertsKeyStoreFile(KeyStoreUtil.java:141) ~[?:?]
	at sun.security.ssl.TrustStoreManager$TrustStoreDescriptor.<clinit>(TrustStoreManager.java:80) ~[?:?]
	at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56) ~[?:?]
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_352]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:279) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:48) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more

I can reproduce on non rhel system by enforcing a cacert file in the jvm.options file which is expected behavior to me.

My questions are:

  • Is this considered a bug? should I open a ticket in elasticsearch repo to add a policy for the system cacert?
  • Is this an error from RHEL and should be fixed in their side?
  • Obviously granting FilePermission read for elasticsearch user (/home/elasticsearch/.java.policy) is an easy workaround, but is it the solution?
1 Like

I have the same happening to a new instance running Centos 9

Version 6.8 is a very long way past EOL. It's completely unsupported and unmaintained at this point in time. Even if this were considered a bug in Elasticsearch, it will not be fixed in such an old version. You must upgrade to a supported version as a matter of urgency.

Furthermore, in all supported versions Elasticsearch recommends running with the bundled JVM rather than using the one the system provides. If you use the bundled JVM you will not have this problem.

1 Like

I see the same problem with elasticsearch 7.9 and this same java version. This system was working until the java upgrade. Log fragment:

[2022-11-22T14:28:08,291][DEBUG][o.e.x.c.s.SSLService ] [GPPES_esserver21] using ssl settings [SSLConfiguration{keyConfig=[NONE], trustConfig=JDK trusted certs], cipherSuites=[[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.3, TLSv1.2, TLSv1.1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}]
[2022-11-22T14:28:08,294][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [GPPES_esserver21] fatal error in thread [main], exiting
java.lang.ExceptionInInitializerError: null
.
.
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.9.3.jar:7.9.3]
.
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")
.

elasticsearch 7.9 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

My previous reply applies here too: 7.9 is long past EOL, and you should be using the bundled JDK.

Rocky Linux 9, Elasticsearch 6.8 for GrayLog.
The same problem.
Resolved by:
1.) put cacerts to /etc/elasticsearch directory (config dir)
2.) edit elasticsearch.yml and add at the end:
xpack.ssl.truststore.path: /etc/elasticsearch/cacerts
xpack.ssl.truststore.password: changeit
reindex.ssl.truststore.path: /etc/elasticsearch/cacerts
reindex.ssl.truststore.password: changeit

Elasticsearch 6.8 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

My previous reply applies here too: 6.8 is long past EOL, and you should be using the bundled JDK.