Service will not start (RHE7)

stacey · August 26, 2021, 2:48pm

Hi

I have installed elasticsearch via the rpm as per the documentation. But I am unable to start the elasticsearch service, but running directly works (with warnings).

ES
7.14.0

JDK (bundled):
openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (build 16.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (build 16.0.1+9, mixed mode, sharing)


Internal exceptions (20 events):
Event: 4.735 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd312da0}: 'void java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bd312da0)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.778 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd430120}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object)'> (0x00000017bd430120)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.905 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd886a60}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object, int, java.lang.Object)'> (0x00000017bd886a60)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.910 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8bfa80}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object, java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd8bfa80)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.911 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8c77a8}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bd8c77a8)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.912 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8cbca0}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bd8cbca0)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.917 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8f9388}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bd8f9388)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.927 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd93d878}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd93d878)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.927 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd941a08}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd941a08)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 5.389 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bde97770}: 'int java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bde97770)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 5.470 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bc149e30}: 'java.lang.Object java.lang.invoke.Invokers$Holder.invokeExact_MT(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bc149e30)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 5.955 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bc94c528}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bc94c528)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.071 Thread 0x00007f319c0254a0 Implicit null exception at 0x00007f318c096c54 to 0x00007f318c097224
Event: 6.200 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb603f80}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bb603f80)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.224 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb69b510}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, long, java.lang.Object)'> (0x00000017bb69b510)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.225 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb6a1b78}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, long)'> (0x00000017bb6a1b78)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.225 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb6a58e8}: 'java.lang.Object java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, long, java.lang.Object)'> (0x00000017bb6a58e8)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.253 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb7970a0}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, int)'> (0x00000017bb7970a0)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.254 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb79b598}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, int)'> (0x00000017bb79b598)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.254 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb79f358}: 'java.lang.Object java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bb79f358)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]

I don't know if this is related, but when I run this directly as the elasticsearch user it works but with warnings:

java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-2615699647026920288/jna10243734774257956315.tmp: /tmp/elasticsearch-2615699647026920288/jna10243734774257956315.tmp: failed to map segment from shared object: Operation not permitted
        at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]
        at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:383) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:227) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:169) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2383) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:746) ~[?:?]
        at java.lang.System.load(System.java:1857) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1019) ~[jna-5.7.0-1.jar:5.7.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:989) ~[jna-5.7.0-1.jar:5.7.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:195) ~[jna-5.7.0-1.jar:5.7.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:375) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:34) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.14.0.jar:7.14.0]

I have added the config to /etc/sysconfig/elasticsearch:
ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp"

The directory exists and is writable, but it appears to be ignored.

Any help would be appreciated.

spinscale · August 26, 2021, 3:30pm

This is the interesting part of the error message

failed to map segment from shared object: Operation not permitted

What permissions is /var/lib/elasticsearch/tmp having?

stacey · August 26, 2021, 4:13pm

Permissions

drwxr-sr-x. 2 elasticsearch elasticsearch xxxx xxxxx tmp

which I assume would be sufficient to allow the user to write to this directory.

Also, just to clarify the “failed to map” is a warning. Elastic search run directly even with this warning.

stacey · August 27, 2021, 10:39am

Okay. I am able to replicate the error (NoSuchMethodError) regardless of how elasticsearch is invoked by setting the ES_TMPDIR.

ES_TMPDIR=/var/lib/elasticsearch/tmp

I know /tmp is noexec (which I don't want to change). Again any help would be appreciated.

stacey · August 27, 2021, 3:03pm

Further developments. I have stripped all the ES_TMP declarations and the JVM arguments are the same for both the service and direct. Directly it works with warnings, but the service fails with the following error:

java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:23) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:22) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:115) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:126) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:971) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:313) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:408) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.0.jar:7.14.0]

I can't understand with the same settings the service fails, but directly it will not.

spinscale · August 30, 2021, 10:07am

oooh, I didnt read the stacktrace carefully enought. The reason why this only happens with the service is, that the SystemdPlugin is not running, when not invoked via the package, because we cannot assume that you use Systemd. It's this code causing problems it seems: elasticsearch/Libsystemd.java at master · elastic/elasticsearch · GitHub

Can you open an issue in the Elasticsearch repository, where you also mention your exact Linux distribution in order to be able to debug this further?

Thanks a lot!

stacey · August 31, 2021, 9:08am

Thanks for the help. I have submitted an issue.

github.com/elastic/elasticsearch

Service failing to start RHEL 7

opened 09:06AM - 31 Aug 21 UTC

passdev-sc

>bug needs:triage

**Elasticsearch version** (`bin/elasticsearch --version`): Version: 7.14.0, Bui…ld: default/rpm/dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1/2021-07-29T20:49:32.864135063Z, JVM: 16.0.1 **Plugins installed**: [] None **JVM version** (`java -version`): _Bundled version is:_ openjdk version "16.0.1" 2021-04-20 OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (build 16.0.1+9) OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (build 16.0.1+9, mixed mode, sharing) _OS Version is:_ openjdk version "16.0.1" 2021-04-20 OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (build 16.0.1+9) OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (build 16.0.1+9, mixed mode, sharing) -- note I have not changed settings, so the bundled version should be the one in use **OS version** (`uname -a` if on a Unix-like system): Linux 1025093 3.10.0-957.21.2.el7.x86_64 #1 SMP Tue May 28 09:26:43 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server 7.9 (Maipo) **Description of the problem including expected versus actual behavior**: After installing elasticsearch via the "Installing from the RPM repository" instructions on https://www.elastic.co/guide/en/elasticsearch/reference/7.14/rpm.html#rpm-repo we are able to start elasticsearch directly /bin/elasticsearch (with warnings). However when attempting to start the service we receive an error. ``` Internal exceptions (20 events): Event: 4.735 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd312da0}: 'void java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bd312da0) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 4.778 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd430120}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object)'> (0x00000017bd430120) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 4.905 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd886a60}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object, int, java.lang.Object)'> (0x00000017bd886a60) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 4.910 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8bfa80}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object, java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd8bfa80) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 4.911 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8c77a8}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bd8c77a8) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 4.912 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8cbca0}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bd8cbca0) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 4.917 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8f9388}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bd8f9388) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 4.927 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd93d878}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd93d878) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 4.927 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd941a08}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd941a08) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 5.389 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bde97770}: 'int java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bde97770) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 5.470 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bc149e30}: 'java.lang.Object java.lang.invoke.Invokers$Holder.invokeExact_MT(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bc149e30) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 5.955 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bc94c528}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bc94c528) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 6.071 Thread 0x00007f319c0254a0 Implicit null exception at 0x00007f318c096c54 to 0x00007f318c097224 Event: 6.200 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb603f80}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bb603f80) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 6.224 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb69b510}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, long, java.lang.Object)'> (0x00000017bb69b510) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 6.225 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb6a1b78}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, long)'> (0x00000017bb6a1b78) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 6.225 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb6a58e8}: 'java.lang.Object java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, long, java.lang.Object)'> (0x00000017bb6a58e8) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 6.253 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb7970a0}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, int)'> (0x00000017bb7970a0) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 6.254 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb79b598}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, int)'> (0x00000017bb79b598) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] Event: 6.254 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb79f358}: 'java.lang.Object java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bb79f358) thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790] ``` Warnings received when starting elastic search directly: ``` java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-2615699647026920288/jna10243734774257956315.tmp: /tmp/elasticsearch-2615699647026920288/jna10243734774257956315.tmp: failed to map segment from shared object: Operation not permitted at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?] at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:383) ~[?:?] at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:227) ~[?:?] at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:169) ~[?:?] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2383) ~[?:?] at java.lang.Runtime.load0(Runtime.java:746) ~[?:?] at java.lang.System.load(System.java:1857) ~[?:?] at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1019) ~[jna-5.7.0-1.jar:5.7.0 (b0)] at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:989) ~[jna-5.7.0-1.jar:5.7.0 (b0)] at com.sun.jna.Native.<clinit>(Native.java:195) ~[jna-5.7.0-1.jar:5.7.0 (b0)] at java.lang.Class.forName0(Native Method) ~[?:?] at java.lang.Class.forName(Class.java:375) ~[?:?] at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:34) [elasticsearch-7.14.0.jar:7.14.0] at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102) [elasticsearch-7.14.0.jar:7.14.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-7.14.0.jar:7.14.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) [elasticsearch-7.14.0.jar:7.14.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.14.0.jar:7.14.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.14.0.jar:7.14.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.14.0.jar:7.14.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.14.0.jar:7.14.0] at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.14.0.jar:7.14.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.14.0.jar:7.14.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.14.0.jar:7.14.0] ``` See the link for a discussion regarding this: https://discuss.elastic.co/t/service-will-not-start-rhe7/282591/5 The current thoughts are that the error is caused as elasticsearch cannot assume the SystemdPlugin is running. https://github.com/elastic/elasticsearch/blob/master/modules/systemd/src/main/java/org/elasticsearch/systemd/Libsystemd.java#L21-L26 **Steps to reproduce**: 1. service elasticsearch start

stacey · September 2, 2021, 1:09pm

With the help of the development team I am able to get the service running by setting TMPDIR. Please note the I removed ES_TMPDIR as this seems to be causing issues when running elasticsearch directly (i.e. not from a service).

system · September 30, 2021, 1:09pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch 6.4.0 won't start on RHEL7 Elasticsearch	2	2509	October 22, 2018
Elasticsearch 7.9 doesn't start Elasticsearch	4	2233	September 21, 2020
ES 7.9 doesn't start Elasticsearch	8	559	September 29, 2020
Elasticsearch Service 7.4/7.3 wont start Elasticsearch	3	1522	November 14, 2019
Elasticsearchversion 7.17.9 is not starting - Exception in thread "main" java.lang.RuntimeException: starting java Elasticsearch	6	777	May 23, 2023

Service will not start (RHE7)

Related topics