Java.io.tmpdir

Hello,

My elasticsearch version is 8.5.3-1

I have a hardened RHEL 8 system which requires that /tmp be mounted with the "noexec" option. I have tried to adjust my jvm.options file in /etc/elasticsearch/jvm.options to set
-Djava.io.tmpdir=/usr/share/elasticsearch/tmp.
I have also created a file in /etc/elasticsearch/jvm.options.d/all.options with the same verbiage.
When I temporarily allow /tmp to be mounted as "exec", and run "ps -ef | grep java.io.tmpdir" , I see -Djava.io.tmpdir=/usr/share/elasticsearch/tmp . However, when I remount /tmp as "noexec", elasticsearch will not start.

Any ideas ?

Thank You For Your Help !

Have you seen this thread?

Thanks,

The unit is not clustered (yet) . I have tried .

No success
Thanks For Your Help !

The docs you linked tell you how to solve the problem. Note that they do not say to set java.io.tmpdir.

Thank You David<

I have done "export ES_TMPDIR=/usr/share/elasticsearch/tmp" with no success.

All suggestions are appreciated. The java.io.tmpdir was an attempt to force the system to accept another /tmp path since ps -ef | grep java.io.tmpdir was showing a path to /tmp/elasticsearch....

Thank You For Your Help !

Ok, what do you mean by "no success" exactly? Did you see an error message? It's hard to help without more details.

Hello Mr Turner,

Here is more info about this

[root@server1 kibana]# echo $ES_TMPDIR
/usr/share/elasticsearch/tmp
[root@server1 kibana]# mount -o remount,noexec /tmp
[root@server1 kibana]# systemctl restart elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Partial output of tail /var/log/elasticsearch/elasticsearch.log below

[2023-01-20T13:17:53,139][INFO ][o.e.n.Node ] [server1] stopping ...
[2023-01-20T13:17:54,097][INFO ][o.e.r.s.FileSettingsService] [server1] shutting down watcher thread
[2023-01-20T13:17:54,107][INFO ][o.e.r.s.FileSettingsService] [server1] watcher service stopped
[2023-01-20T13:17:56,437][INFO ][o.e.x.w.WatcherService ] [server1] stopping watch service, reason [shutdown initiated]
[2023-01-20T13:17:56,530][INFO ][o.e.x.w.WatcherLifeCycleService] [server1] watcher has stopped and shutdown
[2023-01-20T13:17:56,769][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [server1] [controller/7658] [Main.cc@176] ML controller exiting
[2023-01-20T13:17:56,779][INFO ][o.e.x.m.p.NativeController] [server1] Native controller process has stopped - no new native processes can be started
[2023-01-20T13:18:00,539][WARN ][o.e.c.InternalClusterInfoService] [server1] failed to retrieve shard stats from node [--t6YfMXQDKwmrHmFqU9Mw]
org.elasticsearch.transport.SendRequestTransportException: [server1][192.168.1.3:9300][indices:monitor/stats[n]]
at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:923) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:890) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) ~[?:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:781) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:727) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:348) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:334) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:258) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:69) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:121) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:130) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:126) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:202) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1271) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1506) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.fetchIndicesStats(InternalClusterInfoService.java:200) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.execute(InternalClusterInfoService.java:188) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService.refreshAsync(InternalClusterInfoService.java:444) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler.lambda$getListener$0(InternalClusterInfoService.java:401) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:825) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:436) ~[elasticsearch-8.5.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: org.elasticsearch.node.NodeClosedException: node closed {server1}{--t6YfMXQDKwmrHmFqU9Mw}{qSzP2lQ3Q0KJX7QQrhOT7Q}{server1}{192.168.1.3}{192.168.1.3:9300}{cdfhilmrstw}{xpack.installed=true, ml.allocated_processors_double=4.0, ml.max_jvm_size=1958739968, ml.allocated_processors=4, ml.machine_memory=3912589312}
... 47 more
[2023-01-20T13:18:00,383][WARN ][o.e.c.InternalClusterInfoService] [server1] failed to retrieve stats for node [--t6YfMXQDKwmrHmFqU9Mw]
org.elasticsearch.transport.SendRequestTransportException: [server1][192.168.1.3:9300][cluster:monitor/nodes/stats[n]]
at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:923) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:890) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) ~[?:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:781) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:727) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:243) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:122) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:39) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:121) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:130) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:126) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:202) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:676) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:774) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.fetchNodeStats(InternalClusterInfoService.java:291) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.execute(InternalClusterInfoService.java:184) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService.refreshAsync(InternalClusterInfoService.java:444) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler.lambda$getListener$0(InternalClusterInfoService.java:401) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:825) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:436) ~[elasticsearch-8.5.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: org.elasticsearch.node.NodeClosedException: node closed {server1}{--t6YfMXQDKwmrHmFqU9Mw}{qSzP2lQ3Q0KJX7QQrhOT7Q}{server1}{192.168.1.3}{192.168.1.3:9300}{cdfhilmrstw}{xpack.installed=true, ml.allocated_processors_double=4.0, ml.max_jvm_size=1958739968, ml.allocated_processors=4, ml.machine_memory=3912589312}
... 46 more
[2023-01-20T13:18:02,294][INFO ][o.e.n.Node ] [server1] stopped
[2023-01-20T13:18:02,295][INFO ][o.e.n.Node ] [server1] closing ...
[2023-01-20T13:18:02,702][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-Country.mmdb]
[2023-01-20T13:18:02,703][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-ASN.mmdb]
[2023-01-20T13:18:02,703][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-City.mmdb]
[2023-01-20T13:18:02,894][INFO ][o.e.n.Node ] [server1] closed
[2023-01-20T13:18:14,974][ERROR][o.e.b.Elasticsearch ] [server1] fatal exception while booting Elasticsearch
java.lang.ExceptionInInitializerError: null
at org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot(JNANatives.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot(Natives.java:57) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:259) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]
Caused by: java.lang.UnsupportedOperationException: Failed to allocate closure
at com.sun.jna.Native.registerMethod(Native Method) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1906) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1775) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1493) ~[jna-5.10.0.jar:?]
at org.elasticsearch.bootstrap.JNACLibrary.(JNACLibrary.java:38) ~[elasticsearch-8.5.3.jar:?]
... 5 more

[root@server1 kibana]# echo $ES_TMPDIR
/usr/share/elasticsearch/tmp
[root@server1 kibana]# mount -o remount,noexec /tmp
[root@server1 kibana]# systemctl restart elasticsearch

Starting Elasticsearch with systemctl means it won't see the environment variables set in your shell. You need to adjust them in the service file instead.

Hello Mr. Turner,

Elasticsearch runs with the /tmp folder mounted as "noexec" now !!

Please see below ...

mount | grep "/tmp"
/dev/mapper/rhel-tmp on /tmp type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

grep 'ES_TMPDIR' /usr/lib/systemd/system/elasticsearch.service
Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp

grep 'ES_TMPDIR' /etc/sysconfig/elasticsearch
ES_TMPDIR=/usr/share/elasticsearch/tmp

Thank You For Your Help !!

Great, good work!

I opened a PR to improve the docs in this area here:

1 Like

Mr. Turner,

Thank You For Your Help !!

1 Like