I have a hardened RHEL 8 system which requires that /tmp be mounted with the "noexec" option. I have tried to adjust my jvm.options file in /etc/elasticsearch/jvm.options to set
-Djava.io.tmpdir=/usr/share/elasticsearch/tmp.
I have also created a file in /etc/elasticsearch/jvm.options.d/all.options with the same verbiage.
When I temporarily allow /tmp to be mounted as "exec", and run "ps -ef | grep java.io.tmpdir" , I see -Djava.io.tmpdir=/usr/share/elasticsearch/tmp . However, when I remount /tmp as "noexec", elasticsearch will not start.
I have done "export ES_TMPDIR=/usr/share/elasticsearch/tmp" with no success.
All suggestions are appreciated. The java.io.tmpdir was an attempt to force the system to accept another /tmp path since ps -ef | grep java.io.tmpdir was showing a path to /tmp/elasticsearch....
[root@server1 kibana]# echo $ES_TMPDIR
/usr/share/elasticsearch/tmp
[root@server1 kibana]# mount -o remount,noexec /tmp
[root@server1 kibana]# systemctl restart elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
Partial output of tail /var/log/elasticsearch/elasticsearch.log below
[2023-01-20T13:17:53,139][INFO ][o.e.n.Node ] [server1] stopping ...
[2023-01-20T13:17:54,097][INFO ][o.e.r.s.FileSettingsService] [server1] shutting down watcher thread
[2023-01-20T13:17:54,107][INFO ][o.e.r.s.FileSettingsService] [server1] watcher service stopped
[2023-01-20T13:17:56,437][INFO ][o.e.x.w.WatcherService ] [server1] stopping watch service, reason [shutdown initiated]
[2023-01-20T13:17:56,530][INFO ][o.e.x.w.WatcherLifeCycleService] [server1] watcher has stopped and shutdown
[2023-01-20T13:17:56,769][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [server1] [controller/7658] [Main.cc@176] ML controller exiting
[2023-01-20T13:17:56,779][INFO ][o.e.x.m.p.NativeController] [server1] Native controller process has stopped - no new native processes can be started
[2023-01-20T13:18:00,539][WARN ][o.e.c.InternalClusterInfoService] [server1] failed to retrieve shard stats from node [--t6YfMXQDKwmrHmFqU9Mw]
org.elasticsearch.transport.SendRequestTransportException: [server1][192.168.1.3:9300][indices:monitor/stats[n]]
at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:923) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:890) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) ~[?:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:781) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:727) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:348) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:334) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:258) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:69) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:121) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:130) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:126) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:202) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1271) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1506) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.fetchIndicesStats(InternalClusterInfoService.java:200) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.execute(InternalClusterInfoService.java:188) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService.refreshAsync(InternalClusterInfoService.java:444) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler.lambda$getListener$0(InternalClusterInfoService.java:401) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:825) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:436) ~[elasticsearch-8.5.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: org.elasticsearch.node.NodeClosedException: node closed {server1}{--t6YfMXQDKwmrHmFqU9Mw}{qSzP2lQ3Q0KJX7QQrhOT7Q}{server1}{192.168.1.3}{192.168.1.3:9300}{cdfhilmrstw}{xpack.installed=true, ml.allocated_processors_double=4.0, ml.max_jvm_size=1958739968, ml.allocated_processors=4, ml.machine_memory=3912589312}
... 47 more
[2023-01-20T13:18:00,383][WARN ][o.e.c.InternalClusterInfoService] [server1] failed to retrieve stats for node [--t6YfMXQDKwmrHmFqU9Mw]
org.elasticsearch.transport.SendRequestTransportException: [server1][192.168.1.3:9300][cluster:monitor/nodes/stats[n]]
at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:923) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:890) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) ~[?:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:781) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:727) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:243) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:122) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:39) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:121) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:130) ~[?:?]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:126) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:104) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:202) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:676) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:774) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.fetchNodeStats(InternalClusterInfoService.java:291) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.execute(InternalClusterInfoService.java:184) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService.refreshAsync(InternalClusterInfoService.java:444) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler.lambda$getListener$0(InternalClusterInfoService.java:401) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:825) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:436) ~[elasticsearch-8.5.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: org.elasticsearch.node.NodeClosedException: node closed {server1}{--t6YfMXQDKwmrHmFqU9Mw}{qSzP2lQ3Q0KJX7QQrhOT7Q}{server1}{192.168.1.3}{192.168.1.3:9300}{cdfhilmrstw}{xpack.installed=true, ml.allocated_processors_double=4.0, ml.max_jvm_size=1958739968, ml.allocated_processors=4, ml.machine_memory=3912589312}
... 46 more
[2023-01-20T13:18:02,294][INFO ][o.e.n.Node ] [server1] stopped
[2023-01-20T13:18:02,295][INFO ][o.e.n.Node ] [server1] closing ...
[2023-01-20T13:18:02,702][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-Country.mmdb]
[2023-01-20T13:18:02,703][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-ASN.mmdb]
[2023-01-20T13:18:02,703][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [server1] evicted [0] entries from cache after reloading database [/usr/share/elasticsearch/tmp/geoip-databases/--t6YfMXQDKwmrHmFqU9Mw/GeoLite2-City.mmdb]
[2023-01-20T13:18:02,894][INFO ][o.e.n.Node ] [server1] closed
[2023-01-20T13:18:14,974][ERROR][o.e.b.Elasticsearch ] [server1] fatal exception while booting Elasticsearch
java.lang.ExceptionInInitializerError: null
at org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot(JNANatives.java:162) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot(Natives.java:57) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:259) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:166) ~[elasticsearch-8.5.3.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.5.3.jar:?]
Caused by: java.lang.UnsupportedOperationException: Failed to allocate closure
at com.sun.jna.Native.registerMethod(Native Method) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1906) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1775) ~[jna-5.10.0.jar:?]
at com.sun.jna.Native.register(Native.java:1493) ~[jna-5.10.0.jar:?]
at org.elasticsearch.bootstrap.JNACLibrary.(JNACLibrary.java:38) ~[elasticsearch-8.5.3.jar:?]
... 5 more
Starting Elasticsearch with systemctl means it won't see the environment variables set in your shell. You need to adjust them in the service file instead.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.