Elasticearch version[8.6.2] initialization error

Elastic Stack Elasticsearch
1

Hi,

When I'm trying to launch Elasticsearch server v 8.6.2 it fails with following error:

[2023-10-09T13:45:34,090][INFO ][o.e.n.Node               ] [DANIELS-HRLP] version[8.6.2], pid[2188], build[zip/2d58d0f136141f03239816a4e360a8d17b6d8f29/2023-02-13T09:35:20.314882762Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.2/19.0.2+7-44]
[2023-10-09T13:45:34,090][INFO ][o.e.n.Node               ] [DANIELS-HRLP] JVM home [C:\elasticsearch\elasticsearch-8.6.2\jdk], using bundled JDK [true]
[2023-10-09T13:45:34,090][INFO ][o.e.n.Node               ] [DANIELS-HRLP] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=C:\Users\DANIEL~1.SCH\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms16241m, -Xmx16241m, -XX:MaxDirectMemorySize=8518631424, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=zip, --module-path=C:\elasticsearch\elasticsearch-8.6.2\lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2023-10-09T13:45:35,009][ERROR][o.e.b.Elasticsearch      ] [DANIELS-HRLP] fatal exception while booting Elasticsearch
java.lang.ExceptionInInitializerError: null
	at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:142) ~[?:?]
	at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:137) ~[?:?]
	at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:100) ~[?:?]
	at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:80) ~[?:?]
	at java.lang.Class.forName0(Native Method) ~[?:?]
	at java.lang.Class.forName(Class.java:390) ~[?:?]
	at java.lang.Class.forName(Class.java:381) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$static$0(S3RepositoryPlugin.java:61) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.<clinit>(S3RepositoryPlugin.java:55) ~[?:?]
	at jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]
	at jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) ~[?:?]
	at jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300) ~[?:?]
	at jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103) ~[?:?]
	at jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:201) ~[?:?]
	at java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:547) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:497) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:611) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:493) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:290) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:159) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:645) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.node.Node.<init>(Node.java:415) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.node.Node.<init>(Node.java:322) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.6.2.jar:?]
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at java.lang.Class.checkMemberAccess(Class.java:3107) ~[?:?]
	at java.lang.Class.getDeclaredFields(Class.java:2422) ~[?:?]
	at com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector._findFields(AnnotatedFieldCollector.java:73) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector.collect(AnnotatedFieldCollector.java:48) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector.collectFields(AnnotatedFieldCollector.java:43) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.AnnotatedClass._fields(AnnotatedClass.java:371) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.AnnotatedClass.fields(AnnotatedClass.java:343) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:518) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:444) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getPropertyMap(POJOPropertiesCollector.java:405) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getProperties(POJOPropertiesCollector.java:247) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.BasicBeanDescription._properties(BasicBeanDescription.java:164) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findProperties(BasicBeanDescription.java:239) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._findCreatorsFromProperties(BasicDeserializerFactory.java:317) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:271) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:222) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:262) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:151) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:415) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:350) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:654) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4956) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4826) ~[jackson-databind.jar:2.15.0]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3703) ~[jackson-databind.jar:2.15.0]
	at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:261) ~[?:?]
	at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:274) ~[?:?]
	at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:347) ~[?:?]
	... 28 more

My elasticsearch.yml is:

network.host: 127.0.0.1
http.port: 9200
xpack.security.enabled: false
xpack.security.http.ssl.enabled: false

Can you please advise what can be a problem here.

Thank you in advance
Daniel

2

Found the solution:

add:

grant {
permission java.lang.RuntimePermission "accessDeclaredMembers";
};

to Elasticsearch Java .../lib/security/default.policy file.

But I'm not sure that this is a correct way to do it.

1 Like
3

Thanks for sharing your solution.

Instead of this, could you upgrade your cluster and check if the problem is still there? Asking because you should not have to change that file when running the server.

4

Thank you for response. You mean upgrade the cluster to latest version?
Actually my current installation worked for more than a month. It stopped working a few days ago. I guess that it is related to some security updates that took place on my computer.

5

Yes. That's what I meant.

Unless Elastic or its built-in JVM have been upgraded, it should not have happened.

6

No, ES built-in JVM wasn't upgraded. I have re-installed ES from scratch several times, but always got that permission error.

7

Same error with last Elasticsearch v 8.10.2:

[2023-10-09T20:49:36,259][INFO ][o.e.n.Node               ] [DANIELS-HRLP] version[8.10.2], pid[6436], build[zip/6d20dd8ce62365be9b1aca96427de4622e970e9e/2023-09-19T08:16:24.564900370Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/20.0.2/20.0.2+9-78]
[2023-10-09T20:49:36,264][INFO ][o.e.n.Node               ] [DANIELS-HRLP] JVM home [C:\elasticsearch\archive\elasticsearch-8.10.2\jdk], using bundled JDK [true]
[2023-10-09T20:49:36,265][INFO ][o.e.n.Node               ] [DANIELS-HRLP] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, -XX:+UseG1GC, -Djava.io.tmpdir=C:\Users\DANIEL~1.SCH\AppData\Local\Temp\elasticsearch, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms16241m, -Xmx16241m, -XX:MaxDirectMemorySize=8518631424, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=zip, --module-path=C:\elasticsearch\archive\elasticsearch-8.10.2\lib, --add-modules=jdk.net, --add-modules=org.elasticsearch.preallocate, -Djdk.module.main=org.elasticsearch.server]
[2023-10-09T20:49:37,580][ERROR][o.e.b.Elasticsearch      ] [DANIELS-HRLP] fatal exception while booting Elasticsearch
java.lang.ExceptionInInitializerError: null
...
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
...
8

Could you share the full logs from the start please?

9

I think that I found the problem. After setting log level to DEBUG, lines like below appear in log file:
examining jar: C:\Program Files (x86)\IBM\WebSphere MQ\java\lib\com.ibm.mqjms.jar

This jar belongs to IBM MQ installation and was taken from CLASSPATH environment variable. I've upgraded IBM MQ from v 9.1.0 to v 9.3.3 a few days ago. After removal of CLASSPATH environment variable ES was launched without errors.

It seems that there is a conflict between ES 8 and IBM MQ 9.3.3. The CLASSPATH environment variable is set by IBM MQ installation and some classes from IBM MQ jars are used by ES 8 instead of from its internal jars.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.