Elasticsearch service getting failed when /tmp is mounted no exec

prat · November 20, 2020, 10:38am

Hi Team,

elasticsearch service is getting failed when /tmp is mounted with noexec, when try to remount it with exec , service is getting started.

Is below link still valid or there is parmenent solution or workaround ?

Q. I would like to know how can we change the path of Elasticsearch logs which its trying to write in /tmp ? (something like '/var/' or '/opt' which is already mounted as exec)

Q. Also can you please tell, if this logs are temporary and elasticsearch service deletes it after some time ? if yes then only it make sense to change path to something like '/var' or '/opt'

Below are the logs.

systemctl status elasticsearch.service

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─startup-timeout.conf
   Active: failed (Result: exit-code) since Fri 2020-11-20 14:52:12 IST; 16min ago
     Docs: http://www.elastic.co
  Process: 48882 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 48882 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/elasticsearch.service

Nov 20 14:51:34 <HOSTNAME> systemd[1]: Starting Elasticsearch...
Nov 20 14:51:38 <HOSTNAME> elasticsearch[48882]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will ...release.
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: ERROR: [1] bootstrap checks failed
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: [1]: system call filters failed to install; check the logs and fix your configuration or disable sy...own risk
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Failed to start Elasticsearch.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Unit elasticsearch.service entered failed state.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service failed.

journalctl -fu elasticsearch

-- Logs begin at Tue 2020-11-17 13:09:05 IST. --
Nov 20 14:51:38 <HOSTNAME> elasticsearch[48882]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: ERROR: [1] bootstrap checks failed
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Failed to start Elasticsearch.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Unit elasticsearch.service entered failed state.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service failed.

prat · November 20, 2020, 10:43am

/var/log/elasticsearch/<cluster-name.log>

[2020-11-20T14:51:53,369][WARN ][o.e.b.Natives            ] [elasticsearch_1] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp: /tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
        at java.lang.System.load(System.java:1870) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:333) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.4.0.jar:7.4.0]
[2020-11-20T14:51:53,380][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot check if running as root because JNA is not available
[2020-11-20T14:51:53,380][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot install system call filter because JNA is not available
[2020-11-20T14:51:53,381][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot register console handler because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_AS because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives            ] [elasticsearch_1] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2020-11-20T14:51:53,612][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch_1] using [1] data paths, mounts [[/opt (/dev/mapper/appvg-lv_opt)]], net usable_space [121.8gb], net total_space [147.5gb], types [ext4]
[2020-11-20T14:51:53,614][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch_1] heap size [5.1gb], compressed ordinary object pointers [true]
[2020-11-20T14:51:53,617][INFO ][o.e.n.Node               ] [elasticsearch_1] node name [elasticsearch_1], node ID [6OyoyA40S3-z6M7nAnCGIA], cluster name [<cluster name>]
[2020-11-20T14:51:53,617][INFO ][o.e.n.Node               ] [elasticsearch_1] version[7.4.0], pid[48882], build[default/rpm/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z], OS[Linux/3.10.0-1127.19.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13/13+33]
[2020-11-20T14:51:53,618][INFO ][o.e.n.Node               ] [elasticsearch_1] JVM home [/usr/share/elasticsearch/jdk]
[2020-11-20T14:51:53,618][INFO ][o.e.n.Node               ] [elasticsearch_1] JVM arguments [-Xms5288m, -Xmx5288m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-9221628356394379624, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Xms5288m, -Xmx5288m, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=2772434944, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-11-20T14:51:55,967][INFO ][o.e.p.PluginsService     ] [elasticsearch_1] loaded module [aggs-matrix-stats]
[2020-11-20T14:51:55,968][INFO ][o.e.p.PluginsService     ] [elasticsearch_1] loaded module [analysis-common]
org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:11,962][ERROR][o.e.x.m.c.c.StatsCollector] [elasticsearch_1] collector [ccr_stats] failed to collect data
java.lang.NullPointerException: null
        at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:11,963][ERROR][o.e.x.m.c.i.IndexStatsCollector] [elasticsearch_1] collector [index-stats] failed to collect data
java.lang.NullPointerException: null
        at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:12,114][INFO ][o.e.t.TransportService   ] [elasticsearch_1] publish_address {<IP of localhost>:5300}, bound_addresses {0.0.0.0:5300}
[2020-11-20T14:52:12,123][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch_1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-11-20T14:52:12,128][ERROR][o.e.b.Bootstrap          ] [elasticsearch_1] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2020-11-20T14:52:12,132][INFO ][o.e.n.Node               ] [elasticsearch_1] stopping ...
[2020-11-20T14:52:12,155][INFO ][o.e.n.Node               ] [elasticsearch_1] stopped
[2020-11-20T14:52:12,156][INFO ][o.e.n.Node               ] [elasticsearch_1] closing ...
[2020-11-20T14:52:12,166][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch_1] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:66) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.close(SystemdPlugin.java:87) ~[?:?]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:86) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.node.Node.close(Node.java:881) ~[elasticsearch-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:62) ~[elasticsearch-core-7.4.0.jar:7.4.0]
        at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:186) ~[elasticsearch-7.4.0.jar:7.4.0]

warkolm · November 23, 2020, 12:10am

That is the solution, yes.

If you want, the documentation you linked to mentions the config setting for that.
You will need to clean that up yourself though.

prat · November 24, 2020, 6:51am

Hi Warkolm,

Thanks for your reply.

Do you know how much big this logs grows in size.

I was looking for this.

It says to edit /etc/sysconfig/elasticsearch and add ES_JAVA_OPTS="-Djava.io.tmpdir=/var/lib/elasticsearch/tmp" and restart Elasticsearch service.

I will give a try and let everyone know.

Thanks,

system · December 22, 2020, 6:51am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.