Elastic Search can't been started without execute rights on /tmp

mscheepers · October 21, 2024, 12:23pm

Hi All,
I noticed when starting the Elasticsearch services on our RHEL 9 machine is failing when it's been installed with RPM's.
The startup is failing as its trying to execute files on the /tmp path. This is by default not possible as it mounted as no-exec.

i found a article what is mentioning the variable ES_TMPDIR but this doesn't do the trick.

i'm running version 8.15.2

Any suggestions?

Michel

Damian_Pfister · October 21, 2024, 1:33pm

Hi Michel,

You mentioned that the ES_TMPDIR route didn't work. Where in this doc did you have challenges?

Thanks
Damian

mscheepers · October 21, 2024, 1:56pm

Hi @Damian_Pfister thanks for responding:

i have edit the systemd override file as mentioned in the doc. But when i start elastic after it is still looking at the /tmp

see here some journalctl log of the startup:

Oct 21 15:53:34 EDR04 systemd-entrypoint[2616]: Oct 21, 2024 3:53:34 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Oct 21 15:53:34 EDR04 systemd-entrypoint[2616]: WARNING: COMPAT locale provider will be removed in a future release
Oct 21 15:53:35 EDR04 systemd-entrypoint[2616]: Failed to load native library:jansi-2.4.0-9fd69cf2dbe4a98f-libjansi.so. The native library file at /tmp/elasticsearch-665951078554207538/jansi-2.4.0-9fd69cf2dbe4a>
Oct 21 15:53:35 EDR04 systemd-entrypoint[2616]: java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-665951078554207538/jansi-2.4.0-9fd69cf2dbe4a98f-libjansi.so: /tmp/elasticsearch-665951078554207538/jansi-2.4.0->
Oct 21 15:53:35 EDR04 systemd-entrypoint[2616]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/my-test-EDR.log
Oct 21 15:53:36 EDR04 systemd-entrypoint[2616]: ERROR: Elasticsearch died while starting up, with exit code 1
Oct 21 15:53:36 EDR04 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE

Damian_Pfister · October 21, 2024, 2:30pm

Hi @mscheepers

Looking at our Configuring system settings doc it mentions either going the /etc/systemd/system/elasticsearch.service.d/override.conf route or sudo systemctl edit elasticsearch. Which did you take? Can you confirm the change was actually applied (with a cat of the file)? Perhaps it is not reading the entry correctly, due to the systemd service not being restarted sudo systemctl daemon-reload?

Perhaps as an alternative you could attempt the -Djna.tmpdir=<path> java parameter option mentioned in Advanced configuration.

DavidTurner · October 21, 2024, 3:10pm

This is not good advice - see these docs for more details on why setting -Djna.tmpdir is insufficient. Setting ${ES_TMPDIR} is the right answer. If ES isn't seeing that environment variable then that's the problem to address.

mscheepers · October 21, 2024, 3:13pm

Hi @Damian_Pfister,

i used the method : Systemctl edit elasticsearch... and indeed reloaded the deamon..

showing the cat:
[root@EDR04 ~]# cat /etc/systemd/system/elasticsearch.service.d/override.conf
Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp
[root@EDR04 ~]#

mscheepers · October 21, 2024, 3:19pm

hi David,
It looks that setting is not been picked up or is not been followed. Also when i try to export the variable manually then its not been followed during startup.

Michel

DavidTurner · October 21, 2024, 3:46pm

Hmm that is strange. Can you try and locate the full stack trace for the java.lang.UnsatisfiedLinkError? It should be in a log somewhere.

mscheepers · October 21, 2024, 5:01pm

Hi David, that is also my assumption that its something strange:

[2024-10-09T16:29:20,848][INFO ][o.e.n.NativeAccess       ] [EDR04] Using native vector library; to disable start with -Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false
[2024-10-09T16:29:21,579][INFO ][o.e.n.NativeAccess       ] [EDR04] Using [jdk] native provider and native methods for [Linux]
[2024-10-09T16:29:21,778][WARN ][o.e.b.Natives            ] [EDR04] unable to load JNA native support library, native methods will be disabled. See https://www.elastic.co/guide/en/elasticsearch/reference/8.15/executable-jna-tmpdir.html
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-10622378458092083345/jna104089939045083595.tmp: /tmp/elasticsearch-10622378458092083345/jna104089939045083595.tmp: failed to map segment from shared object
        at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]
        at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2399) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:852) ~[?:?]
        at java.lang.System.load(System.java:2030) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1045) ~[jna-5.12.1.jar:?]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:1015) ~[jna-5.12.1.jar:?]
        at com.sun.jna.Native.<clinit>(Native.java:221) ~[jna-5.12.1.jar:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) ~[?:?]
        at java.lang.invoke.MethodHandles$Lookup.ensureInitialized(MethodHandles.java:2897) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:37) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:298) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:170) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75) ~[elasticsearch-8.15.2.jar:?]
[2024-10-09T16:29:21,782][WARN ][o.e.b.Natives            ] [EDR04] cannot install system call filter because JNA is not available
[2024-10-09T16:29:21,785][WARN ][o.e.c.f.FileSystemNatives] [EDR04] unable to load JNA native support library, FileSystemNatives methods will be disabled.
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:413) ~[?:?]
        at java.lang.Class.forName(Class.java:404) ~[?:?]
        at org.elasticsearch.common.filesystem.FileSystemNatives.loadJnaProvider(FileSystemNatives.java:37) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.common.filesystem.FileSystemNatives.<clinit>(FileSystemNatives.java:31) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:332) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:170) ~[elasticsearch-8.15.2.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75) ~[elasticsearch-8.15.2.jar:?]
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-10622378458092083345/jna104089939045083595.tmp: /tmp/elasticsearch-10622378458092083345/jna104089939045083595.tmp: failed to map segment from shared object [in thread "main"]
        at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]
        at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197) ~[?:?]
        at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2399) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:852) ~[?:?]
        at java.lang.System.load(System.java:2030) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1045) ~[jna-5.12.1.jar:?]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:1015) ~[jna-5.12.1.jar:?]
        at com.sun.jna.Native.<clinit>(Native.java:221) ~[jna-5.12.1.jar:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]
        at jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) ~[?:?]
"elasticsearch.log" 83L, 8096B

DavidTurner · October 26, 2024, 9:18am

Ok thanks, that really does look as if systemd is not picking up the env var you're trying to set.

Can you share the whole log from Elasticsearch? I'm particularly interested in the line which shows all the JVM command-line args.

DavidTurner · October 26, 2024, 10:55am

Ah:

[root@EDR04 ~]# cat /etc/systemd/system/elasticsearch.service.d/override.conf
Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

You're missing a [Service] line. This file should contain the following:

[Service]
Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

See these docs for more info.

mscheepers · October 27, 2024, 6:43pm

Thanks @DavidTurner that was indeed it.
it's hard to find in the docs..

Thanks a lot again

Topic		Replies	Views
Elasticsearch service getting failed when /tmp is mounted no exec Elasticsearch	4	3038	December 22, 2020
Elasticsearch cannot be started after an upgrade: Temporary file directory [${ES_TMPDIR}] does not exist or is not accessible Elasticsearch	1	1655	March 3, 2020
Fatal exception while booting Elasticsearch Elasticsearch	12	7857	February 2, 2023
Logstash will not start with /tmp mounted noexec Logstash	7	677	February 28, 2023
Unable to start Elastic on RHEL host Elasticsearch	4	5894	April 23, 2020

Elastic Search can't been started without execute rights on /tmp

Related topics