Service will not start (RHE7)

Hi

I have installed elasticsearch via the rpm as per the documentation. But I am unable to start the elasticsearch service, but running directly works (with warnings).

ES
7.14.0

JDK (bundled):
openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (build 16.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (build 16.0.1+9, mixed mode, sharing)


Internal exceptions (20 events):
Event: 4.735 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd312da0}: 'void java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bd312da0)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.778 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd430120}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object)'> (0x00000017bd430120)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.905 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd886a60}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object, int, java.lang.Object)'> (0x00000017bd886a60)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.910 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8bfa80}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object, java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd8bfa80)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.911 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8c77a8}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bd8c77a8)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.912 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8cbca0}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bd8cbca0)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.917 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd8f9388}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bd8f9388)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.927 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd93d878}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd93d878)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 4.927 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bd941a08}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, int, java.lang.Object, java.lang.Object)'> (0x00000017bd941a08)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 5.389 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bde97770}: 'int java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bde97770)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 5.470 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bc149e30}: 'java.lang.Object java.lang.invoke.Invokers$Holder.invokeExact_MT(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bc149e30)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 5.955 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bc94c528}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bc94c528)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.071 Thread 0x00007f319c0254a0 Implicit null exception at 0x00007f318c096c54 to 0x00007f318c097224
Event: 6.200 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb603f80}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000017bb603f80)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.224 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb69b510}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, long, java.lang.Object)'> (0x00000017bb69b510)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.225 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb6a1b78}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, long)'> (0x00000017bb6a1b78)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.225 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb6a58e8}: 'java.lang.Object java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, long, java.lang.Object)'> (0x00000017bb6a58e8)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.253 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb7970a0}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, int)'> (0x00000017bb7970a0)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.254 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb79b598}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, int)'> (0x00000017bb79b598)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]
Event: 6.254 Thread 0x00007f319c0254a0 Exception <a 'java/lang/NoSuchMethodError'{0x00000017bb79f358}: 'java.lang.Object java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x00000017bb79f358)
thrown [./src/hotspot/share/interpreter/linkResolver.cpp, line 790]

I don't know if this is related, but when I run this directly as the elasticsearch user it works but with warnings:

java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-2615699647026920288/jna10243734774257956315.tmp: /tmp/elasticsearch-2615699647026920288/jna10243734774257956315.tmp: failed to map segment from shared object: Operation not permitted
        at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]
        at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:383) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:227) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:169) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2383) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:746) ~[?:?]
        at java.lang.System.load(System.java:1857) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1019) ~[jna-5.7.0-1.jar:5.7.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:989) ~[jna-5.7.0-1.jar:5.7.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:195) ~[jna-5.7.0-1.jar:5.7.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:375) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:34) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.14.0.jar:7.14.0]

I have added the config to /etc/sysconfig/elasticsearch:
ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp"

The directory exists and is writable, but it appears to be ignored.

Any help would be appreciated.

This is the interesting part of the error message

failed to map segment from shared object: Operation not permitted

What permissions is /var/lib/elasticsearch/tmp having?

Permissions

drwxr-sr-x. 2 elasticsearch elasticsearch xxxx xxxxx tmp

which I assume would be sufficient to allow the user to write to this directory.

Also, just to clarify the “failed to map” is a warning. Elastic search run directly even with this warning.

Okay. I am able to replicate the error (NoSuchMethodError) regardless of how elasticsearch is invoked by setting the ES_TMPDIR.

ES_TMPDIR=/var/lib/elasticsearch/tmp

I know /tmp is noexec (which I don't want to change). Again any help would be appreciated.

Further developments. I have stripped all the ES_TMP declarations and the JVM arguments are the same for both the service and direct. Directly it works with warnings, but the service fails with the following error:

java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:23) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:22) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:115) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:126) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:971) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:313) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:408) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.0.jar:7.14.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.0.jar:7.14.0]

I can't understand with the same settings the service fails, but directly it will not.

oooh, I didnt read the stacktrace carefully enought. The reason why this only happens with the service is, that the SystemdPlugin is not running, when not invoked via the package, because we cannot assume that you use Systemd. It's this code causing problems it seems: elasticsearch/Libsystemd.java at master · elastic/elasticsearch · GitHub

Can you open an issue in the Elasticsearch repository, where you also mention your exact Linux distribution in order to be able to debug this further?

Thanks a lot!

Thanks for the help. I have submitted an issue.

1 Like

With the help of the development team I am able to get the service running by setting TMPDIR. Please note the I removed ES_TMPDIR as this seems to be causing issues when running elasticsearch directly (i.e. not from a service).