Elasticsearch process exits abnormally

Elastic Stack Elasticsearch
1

elasticsearch process exits abnormally
system：CentOS Linux release 7.4.1708 (Core)
3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

Mar 18 14:41:51 GL-BPL1001494 systemd: elasticsearch.service: main process exited, code=killed, status=6/ABRT
Mar 18 14:41:51 GL-BPL1001494 systemd: Unit elasticsearch.service entered failed state.
Mar 18 14:41:51 GL-BPL1001494 systemd: elasticsearch.service failed.
hs_err_pid120106.log

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fea10b954dc, pid=120106, tid=0x00007fe8da735700
#
# JRE version: OpenJDK Runtime Environment (8.0_252-b09) (build 1.8.0_252-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.252-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x804dc]  cfree+0x1c
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007fea0811e000):  ConcurrentGCThread [stack: 0x00007fe8da635000,0x00007fe8da736000] [id=120418]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007fe8d8066ff8

Registers:
RAX=0x0000000000000000, RBX=0x00007fdff8295630, RCX=0x0000000000000001, RDX=0x00007fe8d8067000
RSP=0x00007fe8da734c48, RBP=0x00000000ffffffff, RSI=0x00007fe8d8067000, RDI=0x00007fe8d8067000
R8 =0xffffffffffffffff, R9 =0x0000000000000000, R10=0x0000000000000022, R11=0x0000000000000246
R12=0x0000000000000000, R13=0x000000000000001c, R14=0x0000000000000000, R15=0x00007fea0811e000
RIP=0x00007fea10b954dc, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fe8da734c48)
0x00007fe8da734c48:   00007fea10b8d13a 00007fdff8295630
0x00007fe8da734c58:   00007fea10b8c8d5 000000000000001c
0x00007fe8da734c68:   00007fdff8295630 00007fe8da734d20
0x00007fe8da734c78:   00007fea10b8b651 000000000000002b
0x00007fe8da734c88:   00007fdff8295630 0000000000000001
0x00007fe8da734c98:   000000000000001c 000000000000001c
0x00007fe8da734ca8:   00007fea0805c1e0 00007fea0811e000
0x00007fe8da734cb8:   00007fea10b8103d 0000007900000002
0x00007fe8da734cc8:   00007fea080051f0 00007fe8da734d10
0x00007fe8da734cd8:   000000000000001c 00007fe8da734d20
0x00007fe8da734ce8:   00007fea103deeac 0000000000000000
0x00007fe8da734cf8:   00007fea080051f0 00007fea080b2278
0x00007fe8da734d08:   00007fe8da734d20 00007fe8da734d70
0x00007fe8da734d18:   00007fea103e0959 2d33302d31323032
0x00007fe8da734d28:   31343a3431543831 2b3937322e31343a
0x00007fe8da734d38:   00007f0030303830 3fc999999999999a
0x00007fe8da734d48:   93ed24f45810b900 00000000024d9263
0x00007fe8da734d58:   00007fea10aec6e0 00007fea10af66e0
0x00007fe8da734d68:   00007fea0811e7f0 00007fe8da734ed0
0x00007fe8da734d78:   00007fea0ffef985 00007fea08061990
0x00007fe8da734d88:   00000000000003d8 00007fea0811e000
0x00007fe8da734d98:   00007fea0811e7f0 00007fe8da734e10
0x00007fe8da734da8:   00007fea0811ec18 00007fea0811e840
0x00007fe8da734db8:   00007fea08062160 00007fea0811e830
0x00007fe8da734dc8:   413e99a5867c50d2 3fe0000000000000
0x00007fe8da734dd8:   404e42f68fdc65c4 3fe0000000000000
0x00007fe8da734de8:   413e99ab03913c17 40d227adaf5ba2be
0x00007fe8da734df8:   413e99ab01fd0e1e 00007fea10a9eb20
0x00007fe8da734e08:   00007fea0811d400 00007fea0811e000
0x00007fe8da734e18:   00007fea0811ec20 00007fea0811ec60
0x00007fe8da734e28:   00007fea0811ec70 00007fea0811ed48
0x00007fe8da734e38:   00000000000000d8 00007fea0811ee80 

Instructions: (pc=0x00007fea10b954dc)
0x00007fea10b954bc:   1f 44 00 00 48 8b 05 21 ba 33 00 48 8b 00 48 85
0x00007fea10b954cc:   c0 0f 85 bf 00 00 00 48 85 ff 0f 84 b4 00 00 00
0x00007fea10b954dc:   48 8b 47 f8 48 8d 4f f0 a8 02 75 28 a8 04 48 8d
0x00007fea10b954ec:   3d 6f c2 33 00 74 0c 48 89 c8 48 25 00 00 00 fc 

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x00007fdff8295630 is an unknown value
RCX=0x0000000000000001 is an unknown value
RDX=0x00007fe8d8067000 is an unknown value
RSP=0x00007fe8da734c48 is an unknown value
RBP=0x00000000ffffffff is an unknown value
RSI=0x00007fe8d8067000 is an unknown value
RDI=0x00007fe8d8067000 is an unknown value
R8 =0xffffffffffffffff is an unknown value
R9 =0x0000000000000000 is an unknown value
R10=0x0000000000000022 is an unknown value
R11=0x0000000000000246 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x000000000000001c is an unknown value
R14=0x0000000000000000 is an unknown value
R15=0x00007fea0811e000 is an unknown value


Stack: [0x00007fe8da635000,0x00007fe8da736000],  sp=0x00007fe8da734c48,  free space=1023k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x804dc]  cfree+0x1c
2

elasticsearch log

[2021-03-18T14:28:47,225][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2001256] overhead, spent [629ms] collecting in the last [1.2s]
[2021-03-18T14:32:10,880][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2001459] overhead, spent [256ms] collecting in the last [1s]
[2021-03-18T14:32:40,900][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2001489] overhead, spent [376ms] collecting in the last [1s]
[2021-03-18T14:33:11,906][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2001520] overhead, spent [297ms] collecting in the last [1s]
[2021-03-18T14:33:46,042][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2001554] overhead, spent [270ms] collecting in the last [1s]
[2021-03-18T14:34:54,170][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2001622] overhead, spent [279ms] collecting in the last [1s]
[2021-03-18T14:36:59,812][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2001747] overhead, spent [341ms] collecting in the last [1s]
[2021-03-18T14:39:10,513][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2001877] overhead, spent [266ms] collecting in the last [1s]
[2021-03-18T14:41:25,000][INFO ][o.e.m.j.JvmGcMonitorService] [10.0.1.1_node] [gc][2002011] overhead, spent [284ms] collecting in the last [1s]
[2021-03-18T14:42:27,297][INFO ][o.e.e.NodeEnvironment    ] [10.0.1.1_node] using [2] data paths, mounts [[/ssd1 (/dev/nvme0n1), /ssd2 (/dev/nvme1n1)]], net usable_space [4.1tb], net total_space [5.8tb], types [xfs]
[2021-03-18T14:42:27,300][INFO ][o.e.e.NodeEnvironment    ] [10.0.1.1_node] heap size [31gb], compressed ordinary object pointers [true]
[2021-03-18T14:42:27,362][INFO ][o.e.n.Node               ] [10.0.1.1_node] node name [10.0.1.1_node], node ID [E3nPLjQ7TEKkjI4ZroMTzQ]
[2021-03-18T14:42:27,362][INFO ][o.e.n.Node               ] [10.0.1.1_node] version[6.8.1], pid[63458], build[default/rpm/1fad4e1/2019-06-18T13:16:52.517138Z], OS[Linux/3.10.0-693.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_252/25.252-b09]
[2021-03-18T14:42:27,362][INFO ][o.e.n.Node               ] [10.0.1.1_node] JVM arguments [-Xms31g, -Xmx31g, -XX:+UseG1GC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-5262983360642704363, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2021-03-18T14:42:30,564][INFO ][o.e.p.PluginsService     ] [10.0.1.1_node] loaded module [aggs-matrix-stats]
[2021-03-18T14:42:30,564][INFO ][o.e.p.PluginsService     ] [10.0.1.1_node] loaded module [analysis-common]
[2021-03-18T14:42:30,564][INFO ][o.e.p.PluginsService     ] [10.0.1.1_node] loaded module [ingest-common]
[2021-03-18T14:42:30,564][INFO ][o.e.p.PluginsService     ] [10.0.1.1_node] loaded module [ingest-geoip]
[2021-03-18T14:42:30,564][INFO ][o.e.p.PluginsService     ] [10.0.1.1_node] loaded module [ingest-user-agent]
[2021-03-18T14:42:30,564][INFO ][o.e.p.PluginsService     ] [10.0.1.1_node] loaded module [lang-expression]
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.