Elasticsearch Keep Stopping

w0lfxpunk · June 6, 2022, 6:09am

I am running Elasticsearch 8 on a server with 300gb hard drive and 16gb RAM. The Elasticsearch keep stopping in few hours. I tried restarting it using systemctl and got below message

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Mon 2022-06-06 08:02:53 CEST; 33s ago
     Docs: https://www.elastic.co
  Process: 12648 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
 Main PID: 12648 (code=exited, status=143)

Jun 06 08:01:37  systemd[1]: Starting Elasticsearch...
Jun 06 08:02:17  systemd-entrypoint[12648]: Failed to load native library:jansi-2.4.0-1e5c87f34069d30f-libjansi.so. The native library file at /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so is not executable, make sure that the directory is mounted on a partition without the noexec flag, or set the jansi.tmpdir system property to point to a proper location.  osinfo: Linux/x86_64
Jun 06 08:02:17  systemd-entrypoint[12648]: java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so: /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so: failed to map segment from shared object: Operation not permitted
Jun 06 08:02:52  systemd[1]: elasticsearch.service start operation timed out. Terminating.
Jun 06 08:02:53  systemd[1]: Failed to start Elasticsearch.
Jun 06 08:02:53  systemd[1]: Unit elasticsearch.service entered failed state.
Jun 06 08:02:53  systemd[1]: elasticsearch.service failed.

When I clear the /tmp and try to start it, it works again. Please help

warkolm · June 6, 2022, 7:35am

Please check the Elasticsearch logs and post those.

DineshNaik · June 6, 2022, 11:38am

Looks like some permission issue

See if below links help :

github.com/elastic/elasticsearch

elasticsearch fails to start when tmp directory defined

opened 01:41PM - 17 May 16 UTC

closed 10:57AM - 21 Nov 16 UTC

bubo77

feedback_needed :Delivery/Packaging Team:Delivery

**Elasticsearch version**: 2.3.2 **JVM version**: java version "1.8.0_45" Java…(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) **OS version**: Centos 7 SELinux enabled **Description of the problem including expected versus actual behavior**: We have hardened Centos 7 where /tmp is not writeable We have elasticsearch homedir in custom location and we link that custom location into /var/lib/elasticsearch. We define tmp dir path in /etc/sysconfig/elasticserach as # Additional Java OPTS ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp" (we also tried without "" and with java.io.tmpdir=) **Steps to reproduce**: 1. change tmpdir path in sysconfig 2. restart application **Provide logs (if relevant)**: `[2016-05-17 12:31:30,875][WARN ][bootstrap ] unable to load JNA native support library, native methods will be disabled. java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna306419785920339930.tmp: /tmp/jna--1985354563/jna306419785920339930.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(Unknown Source) at java.lang.ClassLoader.loadLibrary(Unknown Source) at java.lang.Runtime.load0(Unknown Source) at java.lang.System.load(Unknown Source) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:761) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736) at com.sun.jna.Native.<clinit>(Native.java:131) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Unknown Source) at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:89) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)` [elasticsearch_dump.txt](https://github.com/elastic/elasticsearch/files/268267/elasticsearch_dump.txt)

github.com/elastic/elasticsearch

LIBFFI_TMPDIR should point somewhere that permits executables

opened 10:10AM - 30 Aug 21 UTC

closed 08:18AM - 15 Nov 21 UTC

DaveCTurner

>enhancement stalled :Delivery/Packaging Team:Delivery

JNA requires a temporary directory which is not mounted `noexec`, but JNA also u…ses `libffi` which itself requires a temporary directory that isn't mounted `noexec`. If none of the usual suspects work, [`libffi` will fall back to `$HOME`](https://github.com/libffi/libffi/blob/ee1263f7d43bd29b15fc72c4d9520a824e8004df/src/closures.c#L702-L707) which will often work, but in general might also not permit executables (or even exist). We override the temporary directory that JNA uses by setting the `java.io.tmpdir` system property to `ES_TMPDIR`. I believe we should do the same thing for `libffi`, by setting the `LIBFFI_TMPDIR` environment variable. We should also document that this can be overridden much as we do for the `jna.tmpdir` system property (which trumps `java.io.tmpdir` if set). Fixing this would, I believe, also fix #73309.

w0lfxpunk · June 6, 2022, 12:19pm

gist.github.com

https://gist.github.com/vishne0/06dd23569f4b5ffcc33f5fa5f1a38341

gistfile1.txt

[2022-06-06T00:53:07,029][WARN ][o.e.t.ThreadPool         ] [] absolute clock went backwards by [736ms/736ms] while timer thread was sleeping
[2022-06-06T01:29:59,111][INFO ][o.e.x.m.MlDailyMaintenanceService] [] triggering scheduled [ML] maintenance tasks
[2022-06-06T01:29:59,315][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [] Deleting expired data
[2022-06-06T01:29:59,586][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [] Successfully deleted [0] unused stats documents
[2022-06-06T01:29:59,601][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [] Completed deletion of expired ML data
[2022-06-06T01:29:59,622][INFO ][o.e.x.m.MlDailyMaintenanceService] [] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2022-06-06T02:01:24,772][INFO ][o.e.m.j.JvmGcMonitorService] [] [gc][166449] overhead, spent [392ms] collecting in the last [1s]
[2022-06-06T03:29:59,092][INFO ][o.e.x.s.SnapshotRetentionTask] [] starting SLM retention snapshot cleanup task
[2022-06-06T03:29:59,122][INFO ][o.e.x.s.SnapshotRetentionTask] [] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2022-06-06T04:00:05,747][INFO ][o.e.m.j.JvmGcMonitorService] [] [gc][173556] overhead, spent [260ms] collecting in the last [1s]

This file has been truncated. show original

warkolm · June 6, 2022, 11:43pm

There's big gaps in the logs, but there's nothing that even shows Elasticsearch is being asked to shut down, so it's really unclear what is happening.

What do your system logs show at these times?

w0lfxpunk · June 7, 2022, 1:57am

gist.github.com

https://gist.github.com/vishne0/d65ed2f69c618d431b97ab9c986ae467

gistfile1.txt

Failed to load native library:jansi-2.4.0-1e5c87f34069d30f-libjansi.so. The native library file at /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so is not executable, make sure that the directory is mounted on a partition without the noexec flag, or set the jansi.tmpdir system property to point to a proper location.  osinfo: Linux/x86_64
Jun  6 08:02:17 vmi525741 systemd-entrypoint: java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so: /tmp/elasticsearch-3886886234635752144/jansi-2.4.0-1e5c87f34069d30f-libjansi.so: failed to map segment from shared object: Operation not permitted
Jun  6 08:02:52 vmi525741 systemd: elasticsearch.service start operation timed out. Terminating.
Jun  6 08:02:53 vmi525741 systemd: Failed to start Elasticsearch.
Jun  6 08:02:53 vmi525741 systemd: Unit elasticsearch.service entered failed state.
Jun  6 08:02:53 vmi525741 systemd: elasticsearch.service failed.

w0lfxpunk · June 7, 2022, 2:16am

Can it because of noexec set on /tmp ?

warkolm · June 7, 2022, 7:13am

It's possible, yes.

w0lfxpunk · June 7, 2022, 9:57am

but removing noexec is not good if we look at it for security reasons.

w0lfxpunk · June 8, 2022, 3:04am

Any pointers for me ?

w0lfxpunk · June 9, 2022, 3:03am

can someone help me please ?

w0lfxpunk · June 15, 2022, 1:43am

Jun 15 02:46:15 vmi525741 kernel: elasticsearch[v invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jun 15 02:46:15 vmi525741 kernel: elasticsearch[v cpuset=/ mems_allowed=0
Jun 15 02:46:15 vmi525741 kernel: CPU: 0 PID: 13570 Comm: elasticsearch[v Kdump: loaded Tainted: G             L ------------   3.10.0-1160.15.2.el7.x86_64 #1

w0lfxpunk · June 15, 2022, 1:44am

seems Elasticsearch getting an OOM ? this is from /var/log/messages but this server has only one website and 6 CPU with 16GB RAM.

Can someone help me why this is happening ?

warkolm · June 15, 2022, 1:47am

Can you share your Elasticsearch logs from that time?
Did you change noexec on tmp?

w0lfxpunk · June 15, 2022, 2:03am

I didn't change noexec on tmp as it's a security risk. Let me get the logs of that time

w0lfxpunk · June 15, 2022, 2:31am

[2022-06-15T00:53:12,040][WARN ][o.e.t.ThreadPool         ] [host.elastic9154.net] absolute clock went backwards by [316ms/316ms] while timer thread was sleeping
[2022-06-15T01:15:49,658][INFO ][o.e.m.j.JvmGcMonitorService] [host.elastic9154.net] [gc][165815] overhead, spent [500ms] collecting in the last [1.3s]
[2022-06-15T01:29:59,342][INFO ][o.e.x.m.MlDailyMaintenanceService] [host.elastic9154.net] triggering scheduled [ML] maintenance tasks
[2022-06-15T01:29:59,498][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [host.elastic9154.net] Deleting expired data
[2022-06-15T01:29:59,715][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [host.elastic9154.net] Successfully deleted [0] unused stats documents
[2022-06-15T01:29:59,717][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [host.elastic9154.net] Completed deletion of expired ML data
[2022-06-15T01:29:59,719][INFO ][o.e.x.m.MlDailyMaintenanceService] [host.elastic9154.net] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2022-06-15T02:37:26,263][INFO ][o.e.m.j.JvmGcMonitorService] [host.elastic9154.net] [gc][170699] overhead, spent [472ms] collecting in the last [1.4s]
[2022-06-15T03:39:37,383][INFO ][o.e.n.Node               ] [host.elastic9154.net] version[8.1.1], pid[22911], build[default/rpm/d0925dd6f22e07b935750420a3155db6e5c58381/2022-03-17T22:01:32.658689558Z], OS[Linux/3.10.0-1160.15.2.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.2/17.0.2+8]
[2022-06-15T03:39:37,418][INFO ][o.e.n.Node               ] [host.elastic9154.net] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-06-15T03:39:37,418][INFO ][o.e.n.Node               ] [host.elastic9154.net] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-15048116905786448875, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djna.tmpdir=/var/lib/elasticsearch/tmp, -Xms7942m, -Xmx7942m, -XX:MaxDirectMemorySize=4164943872, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2022-06-15T03:39:43,476][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [aggs-matrix-stats]
[2022-06-15T03:39:43,477][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [analysis-common]
[2022-06-15T03:39:43,477][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [constant-keyword]
[2022-06-15T03:39:43,477][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [data-streams]
[2022-06-15T03:39:43,478][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [frozen-indices]
[2022-06-15T03:39:43,478][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [ingest-common]
[2022-06-15T03:39:43,478][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [ingest-geoip]
[2022-06-15T03:39:43,478][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [ingest-user-agent]
[2022-06-15T03:39:43,479][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [kibana]
[2022-06-15T03:39:43,479][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [lang-expression]
[2022-06-15T03:39:43,479][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [lang-mustache]
[2022-06-15T03:39:43,479][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [lang-painless]
[2022-06-15T03:39:43,480][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [legacy-geo]
[2022-06-15T03:39:43,480][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [mapper-extras]
[2022-06-15T03:39:43,480][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [mapper-version]
[2022-06-15T03:39:43,481][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [old-lucene-versions]
[2022-06-15T03:39:43,481][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [parent-join]
[2022-06-15T03:39:43,481][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [percolator]
[2022-06-15T03:39:43,482][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [rank-eval]
[2022-06-15T03:39:43,482][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [reindex]
[2022-06-15T03:39:43,482][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [repositories-metering-api]
[2022-06-15T03:39:43,482][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [repository-azure]
[2022-06-15T03:39:43,483][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [repository-encrypted]
[2022-06-15T03:39:43,483][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [repository-gcs]
[2022-06-15T03:39:43,483][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [repository-s3]
[2022-06-15T03:39:43,484][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [repository-url]
[2022-06-15T03:39:43,484][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [runtime-fields-common]
[2022-06-15T03:39:43,484][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [search-business-rules]
[2022-06-15T03:39:43,485][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [searchable-snapshots]
[2022-06-15T03:39:43,485][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [snapshot-based-recoveries]
[2022-06-15T03:39:43,485][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [snapshot-repo-test-kit]
[2022-06-15T03:39:43,486][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [spatial]
[2022-06-15T03:39:43,486][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [systemd]
[2022-06-15T03:39:43,486][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [transform]
[2022-06-15T03:39:43,486][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [transport-netty4]
[2022-06-15T03:39:43,487][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [unsigned-long]
[2022-06-15T03:39:43,487][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [vector-tile]
[2022-06-15T03:39:43,487][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [vectors]
[2022-06-15T03:39:43,487][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [wildcard]
[2022-06-15T03:39:43,487][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-aggregate-metric]
[2022-06-15T03:39:43,488][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-analytics]
[2022-06-15T03:39:43,488][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-async]
[2022-06-15T03:39:43,488][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-async-search]
[2022-06-15T03:39:43,489][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-autoscaling]
[2022-06-15T03:39:43,489][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-ccr]
[2022-06-15T03:39:43,489][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-core]
[2022-06-15T03:39:43,489][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-deprecation]
[2022-06-15T03:39:43,490][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-enrich]
[2022-06-15T03:39:43,490][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-eql]
[2022-06-15T03:39:43,490][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-fleet]
[2022-06-15T03:39:43,490][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-graph]
[2022-06-15T03:39:43,490][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-identity-provider]
[2022-06-15T03:39:43,491][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-ilm]
[2022-06-15T03:39:43,491][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-logstash]
[2022-06-15T03:39:43,491][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-ml]
[2022-06-15T03:39:43,492][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-monitoring]
[2022-06-15T03:39:43,492][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-ql]
[2022-06-15T03:39:43,492][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-rollup]
[2022-06-15T03:39:43,492][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-security]
[2022-06-15T03:39:43,493][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-shutdown]
[2022-06-15T03:39:43,493][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-sql]
[2022-06-15T03:39:43,493][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-stack]
[2022-06-15T03:39:43,493][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-text-structure]
[2022-06-15T03:39:43,494][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-voting-only-node]
[2022-06-15T03:39:43,494][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] loaded module [x-pack-watcher]
[2022-06-15T03:39:43,495][INFO ][o.e.p.PluginsService     ] [host.elastic9154.net] no plugins loaded
[2022-06-15T03:39:43,553][INFO ][o.e.e.NodeEnvironment    ] [host.elastic9154.net] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [165.1gb], net total_space [392.6gb], types [ext4]
[2022-06-15T03:39:43,553][INFO ][o.e.e.NodeEnvironment    ] [host.elastic9154.net] heap size [7.7gb], compressed ordinary object pointers [true]
[2022-06-15T03:39:43,655][INFO ][o.e.n.Node               ] [host.elastic9154.net] node name [host.elastic9154.net], node ID [03qxzIU-Ruql_PZDiyROdw], cluster name [elasticsearch], roles [ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold]
[2022-06-15T03:39:55,476][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [host.elastic9154.net] [controller/23273] [Main.cc@123] controller (64 bit): Version 8.1.1 (Build f5fbdbeecb0643) Copyright (c) 2022 Elasticsearch BV
[2022-06-15T03:39:55,910][INFO ][o.e.x.s.Security         ] [host.elastic9154.net] Security is disabled
[2022-06-15T03:39:57,932][INFO ][o.e.t.n.NettyAllocator   ] [host.elastic9154.net] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-06-15T03:39:57,982][INFO ][o.e.i.r.RecoverySettings ] [host.elastic9154.net] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-06-15T03:39:58,040][INFO ][o.e.d.DiscoveryModule    ] [host.elastic9154.net] using discovery type [multi-node] and seed hosts providers [settings]
[2022-06-15T03:39:59,942][INFO ][o.e.n.Node               ] [host.elastic9154.net] initialized
[2022-06-15T03:39:59,943][INFO ][o.e.n.Node               ] [host.elastic9154.net] starting ...
[2022-06-15T03:39:59,976][INFO ][o.e.x.s.c.f.PersistentCache] [host.elastic9154.net] persistent cache index loaded
[2022-06-15T03:39:59,977][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [host.elastic9154.net] deprecation component started
[2022-06-15T03:40:00,190][INFO ][o.e.t.TransportService   ] [host.elastic9154.net] publish_address {161.97.140.215:9300}, bound_addresses {161.97.140.215:9300}
[2022-06-15T03:40:00,877][INFO ][o.e.b.BootstrapChecks    ] [host.elastic9154.net] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-06-15T03:40:00,884][INFO ][o.e.c.c.Coordinator      ] [host.elastic9154.net] cluster UUID [ivKIIDiZT42Ufy0aJWlBAA]
[2022-06-15T03:40:01,180][INFO ][o.e.c.s.MasterService    ] [host.elastic9154.net] elected-as-master ([1] nodes joined)[{host.elastic9154.net}{03qxzIU-Ruql_PZDiyROdw}{KXaIj9EbSY6QbQ7VZRYB_Q}{161.97.140.215}{161.97.140.215:9300}{cdfhilmrstw} completing election, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 49, version: 707, delta: master node changed {previous [], current [{host.elastic9154.net}{03qxzIU-Ruql_PZDiyROdw}{KXaIj9EbSY6QbQ7VZRYB_Q}{161.97.140.215}{161.97.140.215:9300}{cdfhilmrstw}]}
[2022-06-15T03:40:01,458][INFO ][o.e.c.s.ClusterApplierService] [host.elastic9154.net] master node changed {previous [], current [{host.elastic9154.net}{03qxzIU-Ruql_PZDiyROdw}{KXaIj9EbSY6QbQ7VZRYB_Q}{161.97.140.215}{161.97.140.215:9300}{cdfhilmrstw}]}, term: 49, version: 707, reason: Publication{term=49, version=707}
[2022-06-15T03:40:01,861][INFO ][o.e.h.AbstractHttpServerTransport] [host.elastic9154.net] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2022-06-15T03:40:01,862][INFO ][o.e.n.Node               ] [host.elastic9154.net] started
[2022-06-15T03:40:03,309][WARN ][r.suppressed             ] [host.elastic9154.net] path: /elasticpostdata/_search, params: {index=elasticpostdata}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

w0lfxpunk · June 16, 2022, 1:59am

[type or paste code here](https://gist.github.com/vishne0/ee427234298abab7936cddbc65e3c3ec)

w0lfxpunk · June 16, 2022, 1:59am

This is the latest log when Elasticsearch crashed I am still clue less

w0lfxpunk · June 16, 2022, 2:17am

S=0x00 PREC=0x00 TTL=247 ID=13072 PROTO=TCP SPT=45352 DPT=11000 WINDOW=1024 RES=0x00 SYN URGP=0
Jun 15 02:46:15 hosti525741 kernel: elasticsearch[v invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jun 15 02:46:15 hosti525741 kernel: elasticsearch[v cpuset=/ mems_allowed=0
Jun 15 02:46:15 hosti525741 kernel: CPU: 0 PID: 13570 Comm: elasticsearch[v Kdump: loaded Tainted: G             L ------------   3.10.0-1160.15.2.el7.x86_64 #1
Jun 15 02:46:15 hosti525741 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014
Jun 15 02:46:15 hosti525741 kernel: Call Trace:
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffacd81fba>] dump_stack+0x19/0x1b
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffacd7c8da>] dump_header+0x90/0x229
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffac7065a2>] ? ktime_get_ts64+0x52/0xf0
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffac75dd7f>] ? delayacct_end+0x8f/0xb0
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffac7c227d>] oom_kill_process+0x2cd/0x490
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffac7c1c6d>] ? oom_unkillable_task+0xcd/0x120
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffac7c296a>] out_of_memory+0x31a/0x500
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffacd7d3f7>] __alloc_pages_slowpath+0x5db/0x729
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffac7c8ee6>] __alloc_pages_nodemask+0x436/0x450
Jun 15 02:46:15 hosti525741 kernel: [<ffffffffac818bb8>] alloc_pages_current+0x98/0x110

Jun 15 02:46:15 hosti525741 kernel: Out of memory: Kill process 13097 (java) score 541 or sacrifice child
Jun 15 02:46:15 hosti525741 kernel: Killed process 13424 (controller), UID 989, total-host:54456kB, anon-rss:532kB, file-rss:0kB, shmem-rss:0kB
Jun 15 02:46:15 hosti525741 kernel: elasticsearch[v invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jun 15 02:46:15 hosti525741 kernel: elasticsearch[v cpuset=/ mems_allowed=0
Jun 15 02:46:15 hosti525741 kernel: CPU: 0 PID: 19839 Comm: elasticsearch[v Kdump: loaded Tainted: G             L ------------   3.10.0-1160.15.2.el7.x86_64 #1

w0lfxpunk · June 16, 2022, 2:17am

The message logs seems something is not right with the Kernel ?