Java.io.tmpdir

chuck1 · January 19, 2023, 5:55pm

Hello,

My elasticsearch version is 8.5.3-1

I have a hardened RHEL 8 system which requires that /tmp be mounted with the "noexec" option. I have tried to adjust my jvm.options file in /etc/elasticsearch/jvm.options to set
-Djava.io.tmpdir=/usr/share/elasticsearch/tmp.
I have also created a file in /etc/elasticsearch/jvm.options.d/all.options with the same verbiage.
When I temporarily allow /tmp to be mounted as "exec", and run "ps -ef | grep java.io.tmpdir" , I see -Djava.io.tmpdir=/usr/share/elasticsearch/tmp . However, when I remount /tmp as "noexec", elasticsearch will not start.

Any ideas ?

Thank You For Your Help !

Christian_Dahlqvist · January 19, 2023, 6:01pm

Have you seen this thread?

chuck1 · January 19, 2023, 6:08pm

Thanks,

The unit is not clustered (yet) . I have tried .

No success
Thanks For Your Help !

DavidTurner · January 19, 2023, 10:34pm

The docs you linked tell you how to solve the problem. Note that they do not say to set java.io.tmpdir.

chuck1 · January 20, 2023, 1:20am

Thank You David<

I have done "export ES_TMPDIR=/usr/share/elasticsearch/tmp" with no success.

All suggestions are appreciated. The java.io.tmpdir was an attempt to force the system to accept another /tmp path since ps -ef | grep java.io.tmpdir was showing a path to /tmp/elasticsearch....

Thank You For Your Help !

DavidTurner · January 20, 2023, 6:20am

Ok, what do you mean by "no success" exactly? Did you see an error message? It's hard to help without more details.

chuck1 · January 20, 2023, 2:23pm

Hello Mr Turner,

Here is more info about this

[root@server1 kibana]# echo $ES_TMPDIR
/usr/share/elasticsearch/tmp
[root@server1 kibana]# mount -o remount,noexec /tmp
[root@server1 kibana]# systemctl restart elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Partial output of tail /var/log/elasticsearch/elasticsearch.log below

[2023-01-20T13:17:53,139][INFO ][o.e.n.Node ] [server1] stopping ...
[2023-01-20T13:17:54,097][INFO ][o.e.r.s.FileSettingsService] [server1] shutting down watcher thread
[2023-01-20T13:17:54,107][INFO ][o.e.r.s.FileSettingsService] [server1] watcher service stopped
[2023-01-20T13:17:56,437][INFO ][o.e.x.w.WatcherService ] [server1] stopping watch service, reason [shutdown initiated]
[2023-01-20T13:17:56,530][INFO ][o.e.x.w.WatcherLifeCycleService] [server1] watcher has stopped and shutdown
[2023-01-20T13:17:56,769][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [server1] [controller/7658] [Main.cc@176] ML controller exiting
[2023-01-20T13:17:56,779][INFO ][o.e.x.m.p.NativeController] [server1] Native controller process has stopped - no new native processes can be started
[2023-01-20T13:18:00,539][WARN ][o.e.c.InternalClusterInfoService] [server1] failed to retrieve shard stats from node [--t6YfMXQDKwmrHmFqU9Mw]
org.elasticsearch.transport.SendRequestTransportException: [server1][192.168.1.3:9300][indices:monitor/stats[n]]
at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:923) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:890) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) ~[?:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:781) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:727) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:348) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:334) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:258) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:69) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:121) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:130) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:126) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:202) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1271) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1506) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.fetchIndicesStats(InternalClusterInfoService.java:200) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.execute(InternalClusterInfoService.java:188) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService.refreshAsync(InternalClusterInfoService.java:444) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler.lambda$getListener$0(InternalClusterInfoService.java:401) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:825) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:436) ~[elasticsearch-8.5.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: org.elasticsearch.node.NodeClosedException: node closed {server1}{--t6YfMXQDKwmrHmFqU9Mw}{qSzP2lQ3Q0KJX7QQrhOT7Q}{server1}{192.168.1.3}{192.168.1.3:9300}{cdfhilmrstw}{xpack.installed=true, ml.allocated_processors_double=4.0, ml.max_jvm_size=1958739968, ml.allocated_processors=4, ml.machine_memory=3912589312}
... 47 more
[2023-01-20T13:18:00,383][WARN ][o.e.c.InternalClusterInfoService] [server1] failed to retrieve stats for node [--t6YfMXQDKwmrHmFqU9Mw]
org.elasticsearch.transport.SendRequestTransportException: [server1][192.168.1.3:9300][cluster:monitor/nodes/stats[n]]
at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:923) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:890) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) ~[?:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:781) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:727) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:243) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:122) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:39) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:121) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:130) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:126) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:202) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:676) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:774) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.fetchNodeStats(InternalClusterInfoService.java:291) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.execute(InternalClusterInfoService.java:184) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService.refreshAsync(InternalClusterInfoService.java:444) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler.lambda$getListener$0(InternalClusterInfoService.java:401) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:825) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:436) ~[elasticsearch-8.5.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: org.elasticsearch.node.NodeClosedException: node closed {server1}{--t6YfMXQDKwmrHmFqU9Mw}{qSzP2lQ3Q0KJX7QQrhOT7Q}{server1}{192.168.1.3}{192.168.1.3:9300}{cdfhilmrstw}{xpack.installed=true, ml.allocated_processors_double=4.0, ml.max_jvm_size=1958739968, ml.allocated_processors=4, ml.machine_memory=3912589312}
... 46 more
[2023-01-20T13:18:02,294][INFO ][o.e.n.Node ] [server1] stopped
[2023-01-20T13:18:02,295][INFO ][o.e.n.Node ] [server1] closing ...
[2023-01-20T13:18:02,702][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-Country.mmdb]
[2023-01-20T13:18:02,703][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-ASN.mmdb]
[2023-01-20T13:18:02,703][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-City.mmdb]
[2023-01-20T13:18:02,894][INFO ][o.e.n.Node ] [server1] closed
[2023-01-20T13:18:14,974][ERROR][o.e.b.Elasticsearch ] [server1] fatal exception while booting Elasticsearch
java.lang.ExceptionInInitializerError: null
at org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot(JNANatives.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot(Natives.java:57) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:259) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]
Caused by: java.lang.UnsupportedOperationException: Failed to allocate closure
at com.sun.jna.Native.registerMethod(Native Method) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1906) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1775) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1493) ~[jna-5.10.0.jar:?]
at org.elasticsearch.bootstrap.JNACLibrary.(JNACLibrary.java:38) ~[elasticsearch-8.5.3.jar:?]
... 5 more

DavidTurner · January 20, 2023, 2:42pm

[root@server1 kibana]# echo $ES_TMPDIR
/usr/share/elasticsearch/tmp
[root@server1 kibana]# mount -o remount,noexec /tmp
[root@server1 kibana]# systemctl restart elasticsearch

Starting Elasticsearch with systemctl means it won't see the environment variables set in your shell. You need to adjust them in the service file instead.

chuck1 · January 20, 2023, 9:20pm

Hello Mr. Turner,

Elasticsearch runs with the /tmp folder mounted as "noexec" now !!

Please see below ...

mount | grep "/tmp"
/dev/mapper/rhel-tmp on /tmp type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

grep 'ES_TMPDIR' /usr/lib/systemd/system/elasticsearch.service
Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

grep 'ES_TMPDIR' /etc/sysconfig/elasticsearch
ES_TMPDIR=/usr/share/elasticsearch/tmp

Thank You For Your Help !!

DavidTurner · January 20, 2023, 9:24pm

Great, good work!

I opened a PR to improve the docs in this area here:

github.com/elastic/elasticsearch

Document how to set ES_TMPDIR in the service file

elastic:main ← DaveCTurner:2023-01-20-ES_TMPDIR-unit-file

opened 09:23PM - 20 Jan 23 UTC

DaveCTurner

+11 -0

Today we suggest that users set `ES_TMPDIR` using `export`, which only works if …you're running things directly from the shell. Yet most users encountering `ES_TMPDIR` problems seem to on RHEL and trying to run things via `systemd`, for whom the `export` suggestion doesn't work. This commit adds to the docs a suggestion of how to adjust the `systemd` service file to set the appropriate environment variable. Relates #80651

chuck1 · January 20, 2023, 9:35pm

Mr. Turner,

Thank You For Your Help !!

system · February 17, 2023, 9:36pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash will not start with /tmp mounted noexec Logstash	7	677	February 28, 2023
Elasticsearch service getting failed when /tmp is mounted no exec Elasticsearch	4	3038	December 22, 2020
\Java\jre1.8.0_241\bin\java.exe" -cp "!ES_CLASSPATH!" "org.elasticsearch.tools.launchers.TempDirectory""` was unexpected at this time Elasticsearch	1	663	May 2, 2020
Fatal exception while booting Elasticsearch Elasticsearch	12	7864	February 2, 2023
Cannot change jna file directory from /tmp to new dir \|\| elasticsearch 6.2.3 ver Elasticsearch	1	793	April 25, 2018

Java.io.tmpdir

Related topics