Elastic Search can't been started without execute rights on /tmp

Elastic Stack Elasticsearch
1

Hi All,
I noticed when starting the Elasticsearch services on our RHEL 9 machine is failing when it's been installed with RPM's.
The startup is failing as its trying to execute files on the /tmp path. This is by default not possible as it mounted as no-exec.

i found a article what is mentioning the variable ES_TMPDIR but this doesn't do the trick.

i'm running version 8.15.2

Any suggestions?

Michel

2

Hi Michel,

You mentioned that the ES_TMPDIR route didn't work. Where in this doc did you have challenges?

Thanks
Damian

3

Hi @Damian_Pfister thanks for responding:

i have edit the systemd override file as mentioned in the doc. But when i start elastic after it is still looking at the /tmp

see here some journalctl log of the startup:

Oct 21 15:53:34 EDR04 systemd-entrypoint[2616]: Oct 21, 2024 3:53:34 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Oct 21 15:53:34 EDR04 systemd-entrypoint[2616]: WARNING: COMPAT locale provider will be removed in a future release
Oct 21 15:53:35 EDR04 systemd-entrypoint[2616]: Failed to load native library:jansi-2.4.0-9fd69cf2dbe4a98f-libjansi.so. The native library file at /tmp/elasticsearch-665951078554207538/jansi-2.4.0-9fd69cf2dbe4a>
Oct 21 15:53:35 EDR04 systemd-entrypoint[2616]: java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-665951078554207538/jansi-2.4.0-9fd69cf2dbe4a98f-libjansi.so: /tmp/elasticsearch-665951078554207538/jansi-2.4.0->
Oct 21 15:53:35 EDR04 systemd-entrypoint[2616]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/my-test-EDR.log
Oct 21 15:53:36 EDR04 systemd-entrypoint[2616]: ERROR: Elasticsearch died while starting up, with exit code 1
Oct 21 15:53:36 EDR04 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
4

Hi @mscheepers

Looking at our Configuring system settings doc it mentions either going the /etc/systemd/system/elasticsearch.service.d/override.conf route or sudo systemctl edit elasticsearch. Which did you take? Can you confirm the change was actually applied (with a cat of the file)? Perhaps it is not reading the entry correctly, due to the systemd service not being restarted sudo systemctl daemon-reload?

Perhaps as an alternative you could attempt the -Djna.tmpdir=<path> java parameter option mentioned in Advanced configuration.

5

This is not good advice - see these docs for more details on why setting -Djna.tmpdir is insufficient. Setting ${ES_TMPDIR} is the right answer. If ES isn't seeing that environment variable then that's the problem to address.

6

Hi @Damian_Pfister,

i used the method : Systemctl edit elasticsearch... and indeed reloaded the deamon..

showing the cat:
[root@EDR04 ~]# cat /etc/systemd/system/elasticsearch.service.d/override.conf
Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp
[root@EDR04 ~]#

7

hi David,
It looks that setting is not been picked up or is not been followed. Also when i try to export the variable manually then its not been followed during startup.

Michel

8

Hmm that is strange. Can you try and locate the full stack trace for the java.lang.UnsatisfiedLinkError? It should be in a log somewhere.

9

Hi David, that is also my assumption that its something strange:

[2024-10-09T16:29:20,848][INFO ][o.e.n.NativeAccess       ] [EDR04] Using native vector library; to disable start with -Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false
[2024-10-09T16:29:21,579][INFO ][o.e.n.NativeAccess       ] [EDR04] Using [jdk] native provider and native methods for [Linux]
[2024-10-09T16:29:21,778][WARN ][o.e.b.Natives            ] [EDR04] unable to load JNA native support library, native methods will be disabled. See https://www.elastic.co/guide/en/elasticsearch/reference/8.15/executable-jna-tmpdir.html
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-10622378458092083345/jna104089939045083595.tmp: /tmp/elasticsearch-10622378458092083345/jna104089939045083595.tmp: failed to map segment from shared object
        at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]
        at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2399) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:852) ~[?:?]
        at java.lang.System.load(System.java:2030) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1045) ~[jna-5.12.1.jar:?]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:1015) ~[jna-5.12.1.jar:?]
        at com.sun.jna.Native.<clinit>(Native.java:221) ~[jna-5.12.1.jar:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) ~[?:?]
        at java.lang.invoke.MethodHandles$Lookup.ensureInitialized(MethodHandles.java:2897) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:37) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:298) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:170) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75) ~[elasticsearch-8.15.2.jar:?]
[2024-10-09T16:29:21,782][WARN ][o.e.b.Natives            ] [EDR04] cannot install system call filter because JNA is not available
[2024-10-09T16:29:21,785][WARN ][o.e.c.f.FileSystemNatives] [EDR04] unable to load JNA native support library, FileSystemNatives methods will be disabled.
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:413) ~[?:?]
        at java.lang.Class.forName(Class.java:404) ~[?:?]
        at org.elasticsearch.common.filesystem.FileSystemNatives.loadJnaProvider(FileSystemNatives.java:37) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.common.filesystem.FileSystemNatives.<clinit>(FileSystemNatives.java:31) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:332) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:170) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75) ~[elasticsearch-8.15.2.jar:?]
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-10622378458092083345/jna104089939045083595.tmp: /tmp/elasticsearch-10622378458092083345/jna104089939045083595.tmp: failed to map segment from shared object [in thread "main"]
        at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]
        at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2399) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:852) ~[?:?]
        at java.lang.System.load(System.java:2030) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1045) ~[jna-5.12.1.jar:?]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:1015) ~[jna-5.12.1.jar:?]
        at com.sun.jna.Native.<clinit>(Native.java:221) ~[jna-5.12.1.jar:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) ~[?:?]
"elasticsearch.log" 83L, 8096B
10

Ok thanks, that really does look as if systemd is not picking up the env var you're trying to set.

Can you share the whole log from Elasticsearch? I'm particularly interested in the line which shows all the JVM command-line args.

11

Ah:

[root@EDR04 ~]# cat /etc/systemd/system/elasticsearch.service.d/override.conf
Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

You're missing a [Service] line. This file should contain the following:

[Service]
Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

See these docs for more info.