Elasticsearch is incompatible with new versions of openjdk from redhat

sayfeddine · November 15, 2022, 7:22am

Redhat introduced a new change in their openjdk packages (starting from 1.8.0-345.b01-5.el8_7) for rhel 8.7 where they check for the system cacert automatically when you initialize a new trustmanager object (even without passing any cert).

This leads to a permissiondenied errors on the system cacert file in the Elasticsearch bootstrap chain even if

xpack.security.enabled: false

I even added these for good measures

xpack.security.transport.ssl.enabled: false

Error

[2022-11-09T07:52:26,851][INFO ][o.e.n.Node               ] [odrzse24-storage-node-1] JVM arguments
[-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75,
-XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60,
-Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m,
-Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true,
-XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true,
-Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0,
-Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true,
-Dlog4j2.formatMsgNoLookups=true,
-Djava.io.tmpdir=/tmp/elasticsearch-1598213987235406706,
-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch,
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails,
-XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution,
-XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log,
-XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m,
-Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch,
-Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2022-11-09T07:52:27,096][ERROR][o.e.b.Bootstrap          ] [odrzse24-storage-node-1] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:616) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.23.jar:6.8.23]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more
Caused by: java.lang.ExceptionInInitializerError
	at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56) ~[?:?]
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_352]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:279) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:48) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_352]
	at java.security.AccessController.checkPermission(AccessController.java:886) ~[?:1.8.0_352]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_352]
	at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_352]
	at java.io.File.exists(File.java:825) ~[?:1.8.0_352]
	at sun.security.tools.KeyStoreUtil.getCacertsKeyStoreFile(KeyStoreUtil.java:141) ~[?:?]
	at sun.security.ssl.TrustStoreManager$TrustStoreDescriptor.<clinit>(TrustStoreManager.java:80) ~[?:?]
	at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56) ~[?:?]
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_352]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:279) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:48) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more

I can reproduce on non rhel system by enforcing a cacert file in the jvm.options file which is expected behavior to me.

My questions are:

Is this considered a bug? should I open a ticket in elasticsearch repo to add a policy for the system cacert?
Is this an error from RHEL and should be fixed in their side?
Obviously granting FilePermission read for elasticsearch user (/home/elasticsearch/.java.policy) is an easy workaround, but is it the solution?

orberkovtree · November 19, 2022, 2:39pm

I have the same happening to a new instance running Centos 9

DavidTurner · November 19, 2022, 6:27pm

Version 6.8 is a very long way past EOL. It's completely unsupported and unmaintained at this point in time. Even if this were considered a bug in Elasticsearch, it will not be fixed in such an old version. You must upgrade to a supported version as a matter of urgency.

Furthermore, in all supported versions Elasticsearch recommends running with the bundled JVM rather than using the one the system provides. If you use the bundled JVM you will not have this problem.

tom_hogan · November 23, 2022, 4:38pm

I see the same problem with elasticsearch 7.9 and this same java version. This system was working until the java upgrade. Log fragment:

[2022-11-22T14:28:08,291][DEBUG][o.e.x.c.s.SSLService ] [GPPES_esserver21] using ssl settings [SSLConfiguration{keyConfig=[NONE], trustConfig=JDK trusted certs], cipherSuites=[[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.3, TLSv1.2, TLSv1.1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}]
[2022-11-22T14:28:08,294][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [GPPES_esserver21] fatal error in thread [main], exiting
java.lang.ExceptionInInitializerError: null
.
.
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.9.3.jar:7.9.3]
.
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")
.

system · November 23, 2022, 4:38pm

elasticsearch 7.9 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

DavidTurner · November 23, 2022, 4:57pm

My previous reply applies here too: 7.9 is long past EOL, and you should be using the bundled JDK.

meesha81 · November 28, 2022, 6:35pm

Rocky Linux 9, Elasticsearch 6.8 for GrayLog.
The same problem.
Resolved by:
1.) put cacerts to /etc/elasticsearch directory (config dir)
2.) edit elasticsearch.yml and add at the end:
xpack.ssl.truststore.path: /etc/elasticsearch/cacerts
xpack.ssl.truststore.password: changeit
reindex.ssl.truststore.path: /etc/elasticsearch/cacerts
reindex.ssl.truststore.password: changeit

system · November 28, 2022, 6:35pm

Elasticsearch 6.8 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

DavidTurner · November 28, 2022, 9:21pm

My previous reply applies here too: 6.8 is long past EOL, and you should be using the bundled JDK.

tlopsm · December 14, 2022, 12:06pm

I have similar experience now. everything was working until current patch cycle

openjdk version "1.8.0_352"
OpenJDK Runtime Environment (build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (build 25.352-b08, mixed mode)

[2022-12-13T15:10:46,552][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [S10.NODE01_esserver1] fatal error in thread [main], exiting
java.lang.ExceptionInInitializerError: null
at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56) ~[?:?]
at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_352]
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:295) ~[?:?]
at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:48) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:437) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1128) ~[?:1.8.0_352]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1290) ~[?:1.8.0_352]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1507) ~[?:1.8.0_352]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:142) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:411) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:274) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$14(Node.java:522) ~[elasticsearch-7.9.3.jar:7.9.3]

I know the ES version is old

Furthermore, in all supported versions Elasticsearch recommends running with the bundled JVM rather than using the one the system provides. If you use the bundled JVM you will not have this problem.

tlopsm · December 15, 2022, 11:15am

Problem solved by installing another version open-jdk java.

DavidTurner · December 17, 2022, 1:55pm

Once again, my previous reply applies here: 7.9 is long past EOL, and you should be using the bundled JDK.

I'm going to lock this thread because there's little point in my repeating the same advice over and over again.

DavidTurner · December 17, 2022, 1:57pm

This topic was automatically closed after 2 minutes. New replies are no longer allowed.