Elasticsearch service getting failed when /tmp is mounted no exec

Hi Team,

elasticsearch service is getting failed when /tmp is mounted with noexec, when try to remount it with exec , service is getting started.

Is below link still valid or there is parmenent solution or workaround ?

Q. I would like to know how can we change the path of Elasticsearch logs which its trying to write in /tmp ? (something like '/var/' or '/opt' which is already mounted as exec)

Q. Also can you please tell, if this logs are temporary and elasticsearch service deletes it after some time ? if yes then only it make sense to change path to something like '/var' or '/opt'

Below are the logs.

systemctl status elasticsearch.service

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─startup-timeout.conf
   Active: failed (Result: exit-code) since Fri 2020-11-20 14:52:12 IST; 16min ago
     Docs: http://www.elastic.co
  Process: 48882 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 48882 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/elasticsearch.service

Nov 20 14:51:34 <HOSTNAME> systemd[1]: Starting Elasticsearch...
Nov 20 14:51:38 <HOSTNAME> elasticsearch[48882]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will ...release.
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: ERROR: [1] bootstrap checks failed
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: [1]: system call filters failed to install; check the logs and fix your configuration or disable sy...own risk
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Failed to start Elasticsearch.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Unit elasticsearch.service entered failed state.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service failed.

journalctl -fu elasticsearch

-- Logs begin at Tue 2020-11-17 13:09:05 IST. --
Nov 20 14:51:38 <HOSTNAME> elasticsearch[48882]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: ERROR: [1] bootstrap checks failed
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Failed to start Elasticsearch.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Unit elasticsearch.service entered failed state.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service failed.
  1. /var/log/elasticsearch/<cluster-name.log>
[2020-11-20T14:51:53,369][WARN ][o.e.b.Natives            ] [elasticsearch_1] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp: /tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
        at java.lang.System.load(System.java:1870) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:333) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.4.0.jar:7.4.0]
[2020-11-20T14:51:53,380][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot check if running as root because JNA is not available
[2020-11-20T14:51:53,380][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot install system call filter because JNA is not available
[2020-11-20T14:51:53,381][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot register console handler because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_AS because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2020-11-20T14:51:53,612][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch_1] using [1] data paths, mounts [[/opt (/dev/mapper/appvg-lv_opt)]], net usable_space [121.8gb], net total_space [147.5gb], types [ext4]
[2020-11-20T14:51:53,614][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch_1] heap size [5.1gb], compressed ordinary object pointers [true]
[2020-11-20T14:51:53,617][INFO ][o.e.n.Node               ] [elasticsearch_1] node name [elasticsearch_1], node ID [6OyoyA40S3-z6M7nAnCGIA], cluster name [<cluster name>]
[2020-11-20T14:51:53,617][INFO ][o.e.n.Node               ] [elasticsearch_1] version[7.4.0], pid[48882], build[default/rpm/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z], OS[Linux/3.10.0-1127.19.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13/13+33]
[2020-11-20T14:51:53,618][INFO ][o.e.n.Node               ] [elasticsearch_1] JVM home [/usr/share/elasticsearch/jdk]
[2020-11-20T14:51:53,618][INFO ][o.e.n.Node               ] [elasticsearch_1] JVM arguments [-Xms5288m, -Xmx5288m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-9221628356394379624, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Xms5288m, -Xmx5288m, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=2772434944, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-11-20T14:51:55,967][INFO ][o.e.p.PluginsService     ] [elasticsearch_1] loaded module [aggs-matrix-stats]
[2020-11-20T14:51:55,968][INFO ][o.e.p.PluginsService     ] [elasticsearch_1] loaded module [analysis-common]
org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:11,962][ERROR][o.e.x.m.c.c.StatsCollector] [elasticsearch_1] collector [ccr_stats] failed to collect data
java.lang.NullPointerException: null
        at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:11,963][ERROR][o.e.x.m.c.i.IndexStatsCollector] [elasticsearch_1] collector [index-stats] failed to collect data
java.lang.NullPointerException: null
        at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:12,114][INFO ][o.e.t.TransportService   ] [elasticsearch_1] publish_address {<IP of localhost>:5300}, bound_addresses {0.0.0.0:5300}
[2020-11-20T14:52:12,123][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch_1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-11-20T14:52:12,128][ERROR][o.e.b.Bootstrap          ] [elasticsearch_1] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2020-11-20T14:52:12,132][INFO ][o.e.n.Node               ] [elasticsearch_1] stopping ...
[2020-11-20T14:52:12,155][INFO ][o.e.n.Node               ] [elasticsearch_1] stopped
[2020-11-20T14:52:12,156][INFO ][o.e.n.Node               ] [elasticsearch_1] closing ...
[2020-11-20T14:52:12,166][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch_1] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:66) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.close(SystemdPlugin.java:87) ~[?:?]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:86) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.node.Node.close(Node.java:881) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:62) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:186) ~[elasticsearch-7.4.0.jar:7.4.0]

That is the solution, yes.

If you want, the documentation you linked to mentions the config setting for that.
You will need to clean that up yourself though.

Hi Warkolm,

Thanks for your reply.

Do you know how much big this logs grows in size.

I was looking for this.

It says to edit /etc/sysconfig/elasticsearch and add ES_JAVA_OPTS="-Djava.io.tmpdir=/var/lib/elasticsearch/tmp" and restart Elasticsearch service.

I will give a try and let everyone know.

Thanks,