# Elasticsearch is incompatible with new versions of openjdk from redhat

**URL:** https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928
**Category:** Elasticsearch
**Created:** [November 15, 2022, 7:22am UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928 "2022-11-15T07:22:21Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![sayfeddine](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sayfeddine/32/99563_2.png) [@sayfeddine](https://discuss.elastic.co/u/sayfeddine)
#### Post date: [November 15, 2022, 7:22am UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/1 "2022-11-15T07:22:21Z")

</div>

Redhat introduced a new change in their openjdk packages (starting from 1.8.0-345.b01-5.el8\_7) for rhel 8.7 where they check for the system cacert automatically when you initialize a new `trustmanager` object (even without passing any cert).

This leads to a `permissiondenied` errors on the system cacert file in the Elasticsearch bootstrap chain even if

```auto
xpack.security.enabled: false

```

I even added these for good measures

```auto
xpack.security.transport.ssl.enabled: false

```

Error

```auto
[2022-11-09T07:52:26,851][INFO][o.e.n.Node] [odrzse24-storage-node-1] JVM arguments
[-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75,
-XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60,
-Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m,
-Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true,
-XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true,
-Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0,
-Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true,
-Dlog4j2.formatMsgNoLookups=true,
-Djava.io.tmpdir=/tmp/elasticsearch-1598213987235406706,
-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch,
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails,
-XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution,
-XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log,
-XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m,
-Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch,
-Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2022-11-09T07:52:27,096][ERROR][o.e.b.Bootstrap] [odrzse24-storage-node-1] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:616) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:558) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.23.jar:6.8.23]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.23.jar:6.8.23]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.23.jar:6.8.23]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more
Caused by: java.lang.ExceptionInInitializerError
	at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56) ~[?:?]
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_352]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:279) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:48) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_352]
	at java.security.AccessController.checkPermission(AccessController.java:886) ~[?:1.8.0_352]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_352]
	at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_352]
	at java.io.File.exists(File.java:825) ~[?:1.8.0_352]
	at sun.security.tools.KeyStoreUtil.getCacertsKeyStoreFile(KeyStoreUtil.java:141) ~[?:?]
	at sun.security.ssl.TrustStoreManager$TrustStoreDescriptor.<clinit>(TrustStoreManager.java:80) ~[?:?]
	at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56) ~[?:?]
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_352]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:279) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:48) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_352]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.8.23.jar:6.8.23]
	... 15 more

```

I can reproduce on non rhel system by enforcing a cacert file in the `jvm.options` file which is expected behavior to me.

My questions are:

- Is this considered a bug? should I open a ticket in elasticsearch repo to add a policy for the system cacert?
- Is this an error from RHEL and should be fixed in their side?
- Obviously granting `FilePermission` `read` for elasticsearch user (`/home/elasticsearch/.java.policy`) is an easy workaround, but is it **the** solution?

---

<div class="post-metadata">

### Author: ![orberkovtree](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orberkovtree/32/91243_2.png) [@orberkovtree](https://discuss.elastic.co/u/orberkovtree)
#### Post date: [November 19, 2022, 2:39pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/2 "2022-11-19T14:39:28Z")

</div>

I have the same happening to a new instance running Centos 9

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [November 19, 2022, 6:27pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/3 "2022-11-19T18:27:43Z")

</div>

Version 6.8 is [a very long way past EOL](https://www.elastic.co/support/eol). It's completely unsupported and unmaintained at this point in time. Even if this were considered a bug in Elasticsearch, it will not be fixed in such an old version. You must upgrade to a supported version as a matter of urgency.

Furthermore, in all supported versions Elasticsearch [recommends](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html#jvm-version) running with the bundled JVM rather than using the one the system provides. If you use the bundled JVM you will not have this problem.

---

<div class="post-metadata">

### Author: ![tom\_hogan](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@tom\_hogan](https://discuss.elastic.co/u/tom_hogan)
#### Post date: [November 23, 2022, 4:38pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/4 "2022-11-23T16:38:53Z")

</div>

I see the same problem with elasticsearch 7.9 and this same java version. This system was working until the java upgrade. Log fragment:

[2022-11-22T14:28:08,291][DEBUG][o.e.x.c.s.SSLService] [GPPES\_esserver21] using ssl settings [SSLConfiguration{keyConfig=[NONE], trustConfig=JDK trusted certs], cipherSuites=[[TLS\_AES\_256\_GCM\_SHA384, TLS\_AES\_128\_GCM\_SHA256, TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_CBC\_SHA384, TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_CBC\_SHA256, TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA384, TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA256, TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_CBC\_SHA, TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_CBC\_SHA, TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA, TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA, TLS\_RSA\_WITH\_AES\_256\_CBC\_SHA256, TLS\_RSA\_WITH\_AES\_128\_CBC\_SHA256, TLS\_RSA\_WITH\_AES\_256\_CBC\_SHA, TLS\_RSA\_WITH\_AES\_128\_CBC\_SHA]], supportedProtocols=[[TLSv1.3, TLSv1.2, TLSv1.1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}]  
[2022-11-22T14:28:08,294][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [GPPES\_esserver21] fatal error in thread [main], exiting  
java.lang.ExceptionInInitializerError: null  
.  
.  
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.9.3.jar:7.9.3]  
.  
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")  
.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 23, 2022, 4:38pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/5 "2022-11-23T16:38:53Z")

</div>

elasticsearch 7.9 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [November 23, 2022, 4:57pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/6 "2022-11-23T16:57:38Z")

</div>

> [@tom\_hogan](#):
>
> I see the same problem with elasticsearch 7.9 and this same java version. This system was working until the java upgrade.

My previous reply applies here too: 7.9 is long past EOL, and you should be using the bundled JDK.

---

<div class="post-metadata">

### Author: ![meesha81](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meesha81/32/113871_2.png) [@meesha81](https://discuss.elastic.co/u/meesha81)
#### Post date: [November 28, 2022, 6:35pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/7 "2022-11-28T18:35:25Z")

</div>

Rocky Linux 9, Elasticsearch 6.8 for GrayLog.  
The same problem.  
Resolved by:  
1.) put cacerts to /etc/elasticsearch directory (config dir)  
2.) edit elasticsearch.yml and add at the end:  
xpack.ssl.truststore.path: /etc/elasticsearch/cacerts  
xpack.ssl.truststore.password: changeit  
reindex.ssl.truststore.path: /etc/elasticsearch/cacerts  
reindex.ssl.truststore.password: changeit

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 28, 2022, 6:35pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/8 "2022-11-28T18:35:26Z")

</div>

Elasticsearch 6.8 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [November 28, 2022, 9:21pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/9 "2022-11-28T21:21:27Z")

</div>

> [@meesha81](#):
>
> Rocky Linux 9, Elasticsearch 6.8 for GrayLog.  
> The same problem.

My previous reply applies here too: 6.8 is long past EOL, and you should be using the bundled JDK.

---

<div class="post-metadata">

### Author: ![tlopsm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlopsm/32/114697_2.png) [@tlopsm](https://discuss.elastic.co/u/tlopsm)
#### Post date: [December 14, 2022, 12:06pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/11 "2022-12-14T12:06:07Z")

</div>

I have similar experience now. everything was working until current patch cycle

openjdk version "1.8.0\_352"  
OpenJDK Runtime Environment (build 1.8.0\_352-b08)  
OpenJDK 64-Bit Server VM (build 25.352-b08, mixed mode)

> [2022-12-13T15:10:46,552][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [S10.NODE01\_esserver1] fatal error in thread [main], exiting  
> java.lang.ExceptionInInitializerError: null  
> at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:56) ~[?:?]  
> at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]  
> at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0\_352]  
> at org.elasticsearch.xpack.core.ssl.CertParsingUtils.trustManager(CertParsingUtils.java:295) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.DefaultJDKTrustConfig.createTrustManager(DefaultJDKTrustConfig.java:48) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:437) ~[?:?]  
> at java.util.HashMap.computeIfAbsent(HashMap.java:1128) ~[?:1.8.0\_352]  
> at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]  
> at java.util.HashMap.forEach(HashMap.java:1290) ~[?:1.8.0\_352]  
> at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1507) ~[?:1.8.0\_352]  
> at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]  
> at org.elasticsearch.xpack.core.ssl.SSLService.(SSLService.java:142) ~[?:?]  
> at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:411) ~[?:?]  
> at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:274) ~[?:?]  
> at org.elasticsearch.node.Node.lambda$new$14(Node.java:522) ~[elasticsearch-7.9.3.jar:7.9.3]

I know the ES version is old

Furthermore, in all supported versions Elasticsearch [recommends](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html#jvm-version) running with the bundled JVM rather than using the one the system provides. If you use the bundled JVM you will not have this problem.

---

<div class="post-metadata">

### Author: ![tlopsm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlopsm/32/114697_2.png) [@tlopsm](https://discuss.elastic.co/u/tlopsm)
#### Post date: [December 15, 2022, 11:15am UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/13 "2022-12-15T11:15:20Z")

</div>

Problem solved by installing another version open-jdk java.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 17, 2022, 1:55pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/14 "2022-12-17T13:55:54Z")

</div>

Once again, my previous reply applies here: 7.9 is long past EOL, and you should be using the bundled JDK.

I'm going to lock this thread because there's little point in my repeating the same advice over and over again.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 17, 2022, 1:57pm UTC](https://discuss.elastic.co/t/elasticsearch-is-incompatible-with-new-versions-of-openjdk-from-redhat/318928/15 "2022-12-17T13:57:47Z")

</div>

This topic was automatically closed after 2 minutes. New replies are no longer allowed.