# Elasticsearch Keep Stopping **URL:** \ **Category:** Elasticsearch\ **Created:** [June 6, 2022, 6:09am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422 "2022-06-06T06:09:28Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 6, 2022, 6:09am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/1 "2022-06-06T06:09:29Z")
I am running Elasticsearch 8 on a server with 300gb hard drive and 16gb RAM. The Elasticsearch keep stopping in few hours. I tried restarting it using systemctl and got below message ```auto ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Mon 2022-06-06 08:02:53 CEST; 33s ago Docs: https://www.elastic.co Process: 12648 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143) Main PID: 12648 (code=exited, status=143) Jun 06 08:01:37 systemd[1]: Starting Elasticsearch... Jun 06 08:02:17 systemd-entrypoint[12648]: Failed to load native library:jansi-2.4.0-1e5c87f34069d30f-libjansi.so. The native library file at /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so is not executable, make sure that the directory is mounted on a partition without the noexec flag, or set the jansi.tmpdir system property to point to a proper location. osinfo: Linux/x86_64 Jun 06 08:02:17 systemd-entrypoint[12648]: java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so: /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so: failed to map segment from shared object: Operation not permitted Jun 06 08:02:52 systemd[1]: elasticsearch.service start operation timed out. Terminating. Jun 06 08:02:53 systemd[1]: Failed to start Elasticsearch. Jun 06 08:02:53 systemd[1]: Unit elasticsearch.service entered failed state. Jun 06 08:02:53 systemd[1]: elasticsearch.service failed. ``` When I clear the /tmp and try to start it, it works again. Please help ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 6, 2022, 7:35am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/2 "2022-06-06T07:35:11Z")
Please check the Elasticsearch logs and post those. ---
**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\ **Post date:** [June 6, 2022, 11:38am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/3 "2022-06-06T11:38:53Z")
Looks like some permission issue > [@w0lfxpunk](#): > > `java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so: /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so: failed to map segment from shared object: Operation not permitted` See if below links help : > > > \*\*Elasticsearch version\*\*: 2.3.2 > > \*\*JVM version\*\*: java version "1.8.0\_45" > Java…(TM) SE Runtime Environment (build 1.8.0\_45-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) > > \*\*OS version\*\*: Centos 7 SELinux enabled > > \*\*Description of the problem including expected versus actual behavior\*\*: > We have hardened Centos 7 where /tmp is not writeable > We have elasticsearch homedir in custom location and we link that custom location into /var/lib/elasticsearch. > We define tmp dir path in /etc/sysconfig/elasticserach as > \# Additional Java OPTS > > ES\_JAVA\_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp" (we also tried without "" and with java.io.tmpdir=) > > \*\*Steps to reproduce\*\*: > 1. change tmpdir path in sysconfig > 2. restart application > > \*\*Provide logs (if relevant)\*\*: > \`\[2016-05-17 12:31:30,875\]\[WARN \]\[bootstrap \] unable to load JNA native support library, native methods will be disabled. > java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna306419785920339930.tmp: /tmp/jna--1985354563/jna306419785920339930.tmp: failed to map segment from shared object: Operation not permitted > at java.lang.ClassLoader$NativeLibrary.load(Native Method) > at java.lang.ClassLoader.loadLibrary0(Unknown Source) > at java.lang.ClassLoader.loadLibrary(Unknown Source) > at java.lang.Runtime.load0(Unknown Source) > at java.lang.System.load(Unknown Source) > at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:761) > at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736) > at com.sun.jna.Native.\(Native.java:131) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Unknown Source) > at org.elasticsearch.bootstrap.Natives.\(Natives.java:45) > at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:89) > at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144) > at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270) > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)\` > \[elasticsearch\_dump.txt\](https://github.com/elastic/elasticsearch/files/268267/elasticsearch\_dump.txt) > > > JNA requires a temporary directory which is not mounted \`noexec\`, but JNA also u…ses \`libffi\` which itself requires a temporary directory that isn't mounted \`noexec\`. If none of the usual suspects work, \[\`libffi\` will fall back to \`$HOME\`\](https://github.com/libffi/libffi/blob/ee1263f7d43bd29b15fc72c4d9520a824e8004df/src/closures.c#L702-L707) which will often work, but in general might also not permit executables (or even exist). > > We override the temporary directory that JNA uses by setting the \`java.io.tmpdir\` system property to \`ES\_TMPDIR\`. I believe we should do the same thing for \`libffi\`, by setting the \`LIBFFI\_TMPDIR\` environment variable. We should also document that this can be overridden much as we do for the \`jna.tmpdir\` system property (which trumps \`java.io.tmpdir\` if set). > > Fixing this would, I believe, also fix #73309. ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 6, 2022, 12:19pm UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/4 "2022-06-06T12:19:00Z")
> ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 6, 2022, 11:43pm UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/5 "2022-06-06T23:43:21Z")
There's big gaps in the logs, but there's nothing that even shows Elasticsearch is being asked to shut down, so it's really unclear what is happening. What do your system logs show at these times? ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 7, 2022, 1:57am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/6 "2022-06-07T01:57:47Z")
> ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 7, 2022, 2:16am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/7 "2022-06-07T02:16:01Z")
Can it because of noexec set on /tmp ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 7, 2022, 7:13am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/8 "2022-06-07T07:13:19Z")
It's possible, yes. ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 7, 2022, 9:57am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/9 "2022-06-07T09:57:21Z")
but removing noexec is not good if we look at it for security reasons. ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 8, 2022, 3:04am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/10 "2022-06-08T03:04:57Z")
Any pointers for me ? ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 9, 2022, 3:03am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/11 "2022-06-09T03:03:22Z")
can someone help me please ? ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 15, 2022, 1:43am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/12 "2022-06-15T01:43:32Z")
```auto Jun 15 02:46:15 vmi525741 kernel: elasticsearch[v invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Jun 15 02:46:15 vmi525741 kernel: elasticsearch[v cpuset=/ mems_allowed=0 Jun 15 02:46:15 vmi525741 kernel: CPU: 0 PID: 13570 Comm: elasticsearch[v Kdump: loaded Tainted: G L ------------ 3.10.0-1160.15.2.el7.x86_64 #1 ``` ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 15, 2022, 1:44am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/13 "2022-06-15T01:44:52Z")
seems Elasticsearch getting an OOM ? this is from /var/log/messages but this server has only one website and 6 CPU with 16GB RAM. Can someone help me why this is happening ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 15, 2022, 1:47am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/14 "2022-06-15T01:47:19Z")
Can you share your Elasticsearch logs from that time? Did you change noexec on tmp? ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 15, 2022, 2:03am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/15 "2022-06-15T02:03:48Z")
I didn't change noexec on tmp as it's a security risk. Let me get the logs of that time ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 15, 2022, 2:31am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/16 "2022-06-15T02:31:24Z")
```auto [2022-06-15T00:53:12,040][WARN][o.e.t.ThreadPool] [host.elastic9154.net] absolute clock went backwards by [316ms/316ms] while timer thread was sleeping [2022-06-15T01:15:49,658][INFO][o.e.m.j.JvmGcMonitorService] [host.elastic9154.net] [gc][165815] overhead, spent [500ms] collecting in the last [1.3s] [2022-06-15T01:29:59,342][INFO][o.e.x.m.MlDailyMaintenanceService] [host.elastic9154.net] triggering scheduled [ML] maintenance tasks [2022-06-15T01:29:59,498][INFO][o.e.x.m.a.TransportDeleteExpiredDataAction] [host.elastic9154.net] Deleting expired data [2022-06-15T01:29:59,715][INFO][o.e.x.m.j.r.UnusedStatsRemover] [host.elastic9154.net] Successfully deleted [0] unused stats documents [2022-06-15T01:29:59,717][INFO][o.e.x.m.a.TransportDeleteExpiredDataAction] [host.elastic9154.net] Completed deletion of expired ML data [2022-06-15T01:29:59,719][INFO][o.e.x.m.MlDailyMaintenanceService] [host.elastic9154.net] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask [2022-06-15T02:37:26,263][INFO][o.e.m.j.JvmGcMonitorService] [host.elastic9154.net] [gc][170699] overhead, spent [472ms] collecting in the last [1.4s] [2022-06-15T03:39:37,383][INFO][o.e.n.Node] [host.elastic9154.net] version[8.1.1], pid[22911], build[default/rpm/d0925dd6f22e07b935750420a3155db6e5c58381/2022-03-17T22:01:32.658689558Z], OS[Linux/3.10.0-1160.15.2.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.2/17.0.2+8] [2022-06-15T03:39:37,418][INFO][o.e.n.Node] [host.elastic9154.net] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true] [2022-06-15T03:39:37,418][INFO][o.e.n.Node] [host.elastic9154.net] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-15048116905786448875, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djna.tmpdir=/var/lib/elasticsearch/tmp, -Xms7942m, -Xmx7942m, -XX:MaxDirectMemorySize=4164943872, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true] [2022-06-15T03:39:43,476][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [aggs-matrix-stats] [2022-06-15T03:39:43,477][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [analysis-common] [2022-06-15T03:39:43,477][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [constant-keyword] [2022-06-15T03:39:43,477][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [data-streams] [2022-06-15T03:39:43,478][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [frozen-indices] [2022-06-15T03:39:43,478][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [ingest-common] [2022-06-15T03:39:43,478][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [ingest-geoip] [2022-06-15T03:39:43,478][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [ingest-user-agent] [2022-06-15T03:39:43,479][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [kibana] [2022-06-15T03:39:43,479][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [lang-expression] [2022-06-15T03:39:43,479][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [lang-mustache] [2022-06-15T03:39:43,479][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [lang-painless] [2022-06-15T03:39:43,480][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [legacy-geo] [2022-06-15T03:39:43,480][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [mapper-extras] [2022-06-15T03:39:43,480][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [mapper-version] [2022-06-15T03:39:43,481][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [old-lucene-versions] [2022-06-15T03:39:43,481][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [parent-join] [2022-06-15T03:39:43,481][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [percolator] [2022-06-15T03:39:43,482][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [rank-eval] [2022-06-15T03:39:43,482][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [reindex] [2022-06-15T03:39:43,482][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [repositories-metering-api] [2022-06-15T03:39:43,482][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [repository-azure] [2022-06-15T03:39:43,483][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [repository-encrypted] [2022-06-15T03:39:43,483][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [repository-gcs] [2022-06-15T03:39:43,483][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [repository-s3] [2022-06-15T03:39:43,484][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [repository-url] [2022-06-15T03:39:43,484][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [runtime-fields-common] [2022-06-15T03:39:43,484][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [search-business-rules] [2022-06-15T03:39:43,485][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [searchable-snapshots] [2022-06-15T03:39:43,485][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [snapshot-based-recoveries] [2022-06-15T03:39:43,485][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [snapshot-repo-test-kit] [2022-06-15T03:39:43,486][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [spatial] [2022-06-15T03:39:43,486][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [systemd] [2022-06-15T03:39:43,486][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [transform] [2022-06-15T03:39:43,486][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [transport-netty4] [2022-06-15T03:39:43,487][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [unsigned-long] [2022-06-15T03:39:43,487][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [vector-tile] [2022-06-15T03:39:43,487][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [vectors] [2022-06-15T03:39:43,487][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [wildcard] [2022-06-15T03:39:43,487][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-aggregate-metric] [2022-06-15T03:39:43,488][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-analytics] [2022-06-15T03:39:43,488][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-async] [2022-06-15T03:39:43,488][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-async-search] [2022-06-15T03:39:43,489][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-autoscaling] [2022-06-15T03:39:43,489][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-ccr] [2022-06-15T03:39:43,489][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-core] [2022-06-15T03:39:43,489][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-deprecation] [2022-06-15T03:39:43,490][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-enrich] [2022-06-15T03:39:43,490][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-eql] [2022-06-15T03:39:43,490][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-fleet] [2022-06-15T03:39:43,490][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-graph] [2022-06-15T03:39:43,490][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-identity-provider] [2022-06-15T03:39:43,491][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-ilm] [2022-06-15T03:39:43,491][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-logstash] [2022-06-15T03:39:43,491][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-ml] [2022-06-15T03:39:43,492][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-monitoring] [2022-06-15T03:39:43,492][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-ql] [2022-06-15T03:39:43,492][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-rollup] [2022-06-15T03:39:43,492][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-security] [2022-06-15T03:39:43,493][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-shutdown] [2022-06-15T03:39:43,493][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-sql] [2022-06-15T03:39:43,493][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-stack] [2022-06-15T03:39:43,493][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-text-structure] [2022-06-15T03:39:43,494][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-voting-only-node] [2022-06-15T03:39:43,494][INFO][o.e.p.PluginsService] [host.elastic9154.net] loaded module [x-pack-watcher] [2022-06-15T03:39:43,495][INFO][o.e.p.PluginsService] [host.elastic9154.net] no plugins loaded [2022-06-15T03:39:43,553][INFO][o.e.e.NodeEnvironment] [host.elastic9154.net] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [165.1gb], net total_space [392.6gb], types [ext4] [2022-06-15T03:39:43,553][INFO][o.e.e.NodeEnvironment] [host.elastic9154.net] heap size [7.7gb], compressed ordinary object pointers [true] [2022-06-15T03:39:43,655][INFO][o.e.n.Node] [host.elastic9154.net] node name [host.elastic9154.net], node ID [03qxzIU-Ruql_PZDiyROdw], cluster name [elasticsearch], roles [ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold] [2022-06-15T03:39:55,476][INFO][o.e.x.m.p.l.CppLogMessageHandler] [host.elastic9154.net] [controller/23273] [Main.cc@123] controller (64 bit): Version 8.1.1 (Build f5fbdbeecb0643) Copyright (c) 2022 Elasticsearch BV [2022-06-15T03:39:55,910][INFO][o.e.x.s.Security] [host.elastic9154.net] Security is disabled [2022-06-15T03:39:57,932][INFO][o.e.t.n.NettyAllocator] [host.elastic9154.net] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}] [2022-06-15T03:39:57,982][INFO][o.e.i.r.RecoverySettings] [host.elastic9154.net] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b] [2022-06-15T03:39:58,040][INFO][o.e.d.DiscoveryModule] [host.elastic9154.net] using discovery type [multi-node] and seed hosts providers [settings] [2022-06-15T03:39:59,942][INFO][o.e.n.Node] [host.elastic9154.net] initialized [2022-06-15T03:39:59,943][INFO][o.e.n.Node] [host.elastic9154.net] starting ... [2022-06-15T03:39:59,976][INFO][o.e.x.s.c.f.PersistentCache] [host.elastic9154.net] persistent cache index loaded [2022-06-15T03:39:59,977][INFO][o.e.x.d.l.DeprecationIndexingComponent] [host.elastic9154.net] deprecation component started [2022-06-15T03:40:00,190][INFO][o.e.t.TransportService] [host.elastic9154.net] publish_address {161.97.140.215:9300}, bound_addresses {161.97.140.215:9300} [2022-06-15T03:40:00,877][INFO][o.e.b.BootstrapChecks] [host.elastic9154.net] bound or publishing to a non-loopback address, enforcing bootstrap checks [2022-06-15T03:40:00,884][INFO][o.e.c.c.Coordinator] [host.elastic9154.net] cluster UUID [ivKIIDiZT42Ufy0aJWlBAA] [2022-06-15T03:40:01,180][INFO][o.e.c.s.MasterService] [host.elastic9154.net] elected-as-master ([1] nodes joined)[{host.elastic9154.net}{03qxzIU-Ruql_PZDiyROdw}{KXaIj9EbSY6QbQ7VZRYB_Q}{161.97.140.215}{161.97.140.215:9300}{cdfhilmrstw} completing election, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 49, version: 707, delta: master node changed {previous [], current [{host.elastic9154.net}{03qxzIU-Ruql_PZDiyROdw}{KXaIj9EbSY6QbQ7VZRYB_Q}{161.97.140.215}{161.97.140.215:9300}{cdfhilmrstw}]} [2022-06-15T03:40:01,458][INFO][o.e.c.s.ClusterApplierService] [host.elastic9154.net] master node changed {previous [], current [{host.elastic9154.net}{03qxzIU-Ruql_PZDiyROdw}{KXaIj9EbSY6QbQ7VZRYB_Q}{161.97.140.215}{161.97.140.215:9300}{cdfhilmrstw}]}, term: 49, version: 707, reason: Publication{term=49, version=707} [2022-06-15T03:40:01,861][INFO][o.e.h.AbstractHttpServerTransport] [host.elastic9154.net] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200} [2022-06-15T03:40:01,862][INFO][o.e.n.Node] [host.elastic9154.net] started [2022-06-15T03:40:03,309][WARN][r.suppressed] [host.elastic9154.net] path: /elasticpostdata/_search, params: {index=elasticpostdata} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed ``` ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 16, 2022, 1:59am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/17 "2022-06-16T01:59:06Z")
```auto [type or paste code here](https://gist.github.com/vishne0/ee427234298abab7936cddbc65e3c3ec) ``` ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 16, 2022, 1:59am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/18 "2022-06-16T01:59:36Z")
This is the latest log when Elasticsearch crashed I am still clue less ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 16, 2022, 2:17am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/19 "2022-06-16T02:17:14Z")
```auto S=0x00 PREC=0x00 TTL=247 ID=13072 PROTO=TCP SPT=45352 DPT=11000 WINDOW=1024 RES=0x00 SYN URGP=0 Jun 15 02:46:15 hosti525741 kernel: elasticsearch[v invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Jun 15 02:46:15 hosti525741 kernel: elasticsearch[v cpuset=/ mems_allowed=0 Jun 15 02:46:15 hosti525741 kernel: CPU: 0 PID: 13570 Comm: elasticsearch[v Kdump: loaded Tainted: G L ------------ 3.10.0-1160.15.2.el7.x86_64 #1 Jun 15 02:46:15 hosti525741 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014 Jun 15 02:46:15 hosti525741 kernel: Call Trace: Jun 15 02:46:15 hosti525741 kernel: [] dump_stack+0x19/0x1b Jun 15 02:46:15 hosti525741 kernel: [] dump_header+0x90/0x229 Jun 15 02:46:15 hosti525741 kernel: [] ? ktime_get_ts64+0x52/0xf0 Jun 15 02:46:15 hosti525741 kernel: [] ? delayacct_end+0x8f/0xb0 Jun 15 02:46:15 hosti525741 kernel: [] oom_kill_process+0x2cd/0x490 Jun 15 02:46:15 hosti525741 kernel: [] ? oom_unkillable_task+0xcd/0x120 Jun 15 02:46:15 hosti525741 kernel: [] out_of_memory+0x31a/0x500 Jun 15 02:46:15 hosti525741 kernel: [] __alloc_pages_slowpath+0x5db/0x729 Jun 15 02:46:15 hosti525741 kernel: [] __alloc_pages_nodemask+0x436/0x450 Jun 15 02:46:15 hosti525741 kernel: [] alloc_pages_current+0x98/0x110 Jun 15 02:46:15 hosti525741 kernel: Out of memory: Kill process 13097 (java) score 541 or sacrifice child Jun 15 02:46:15 hosti525741 kernel: Killed process 13424 (controller), UID 989, total-host:54456kB, anon-rss:532kB, file-rss:0kB, shmem-rss:0kB Jun 15 02:46:15 hosti525741 kernel: elasticsearch[v invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Jun 15 02:46:15 hosti525741 kernel: elasticsearch[v cpuset=/ mems_allowed=0 Jun 15 02:46:15 hosti525741 kernel: CPU: 0 PID: 19839 Comm: elasticsearch[v Kdump: loaded Tainted: G L ------------ 3.10.0-1160.15.2.el7.x86_64 #1 ``` ---
**Author:** ![w0lfxpunk](https://avatars.discourse-cdn.com/v4/letter/w/73ab20/32.png) [@w0lfxpunk](https://discuss.elastic.co/u/w0lfxpunk)\ **Post date:** [June 16, 2022, 2:17am UTC](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422/20 "2022-06-16T02:17:54Z")
The message logs seems something is not right with the Kernel ? [Next page](https://discuss.elastic.co/t/elasticsearch-keep-stopping/306422.md?page=2)