Elasticsearch fails to start after reboot

Elastic Stack Elasticsearch
1

I am running the following version of Elasticsearch

 "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  }

on

OpenJDK Runtime Environment (Red_Hat-11.0.20.0.8-1) (build 11.0.20+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.20.0.8-1) (build 11.0.20+8-LTS, mixed mode, sharing)

on a virtual machine with AlmaLinux 8.8 (Sapphire Caracal) installed.

Elasticsearch has been configured to run as a service:

systemctl is-enabled elasticsearch.service

says it is enabled.

However, after the virtual machine has been restarted, the elasticsearch service fails to start producing the log below. I am still able to start it manually with

 service elasticsearch restart

but I wonder how to make it start automatically.

2

The error log:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f0c67e0c932 (sent by kill), pid=486, tid=1246
#
# JRE version: OpenJDK Runtime Environment AdoptOpenJDK (15.0.1+9) (build 15.0.1+9)
# Java VM: OpenJDK 64-Bit Server VM AdoptOpenJDK (15.0.1+9, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# J 287 c2 java.io.StreamTokenizer.read()I java.base@15.0.1 (38 bytes) @ 0x00007f0c67e0c932 [0x00007f0c67e0c7a0+0x0000000000000192]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /usr/share/elasticsearch/core.486)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/AdoptOpenJDK/openjdk-support/issues
#

---------------  S U M M A R Y ------------

Command Line: -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseG1GC -Djava.io.tmpdir=/usr/share/elasticsearch/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Xms1024m -Xmx1024m -XX:+UseG1GC -Djava.io.tmpdir=/usr/share/elasticsearch/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=536870912 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet

Host: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, 4 cores, 2G, AlmaLinux release 8.8 (Sapphire Caracal)
Time: Sun Nov  5 15:03:41 2023 UTC elapsed time: 4.660056 seconds (0d 0h 0m 4s)

---------------  T H R E A D  ---------------

Current thread (0x00007f0c7802bf10):  JavaThread "main" [_thread_in_Java, id=1246, stack(0x00007f0c80e72000,0x00007f0c80f73000)]

Stack: [0x00007f0c80e72000,0x00007f0c80f73000],  sp=0x00007f0c80f70538,  free space=1017k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 287 c2 java.io.StreamTokenizer.read()I java.base@15.0.1 (38 bytes) @ 0x00007f0c67e0c932 [0x00007f0c67e0c7a0+0x0000000000000192]

[error occurred during error reporting (printing native stack), id 0xb, SIGSEGV (0xb) at pc=0x00007f0c7f266e64]


siginfo: si_signo: 11 (SIGSEGV), si_code: 0 (SI_USER), si_pid: 0 (invalid), si_uid: 0

Register to memory mapping:

RAX=0x000000000000006b is an unknown value
RBX=0x000000000000006b is an unknown value
RCX=0x000000000000166e is an unknown value
RDX=0x0000000040100001 is an unknown value
RSP=0x00007f0c80f70538 is pointing into the stack for thread: 0x00007f0c7802bf10
RBP=0x00007f0c80f70598 is pointing into the stack for thread: 0x00007f0c7802bf10
RSI=0x00000000ff579d48 is an oop: java.io.StreamTokenizer 
{0x00000000ff579d48} - klass: 'java/io/StreamTokenizer'
RDI=0x00000000c04aee68 is an oop: [B 
{0x00000000c04aee68} - klass: {type array byte}
 - length: 15
R8 =0x00000000ff5737b8 is an oop: java.io.InputStreamReader 
{0x00000000ff5737b8} - klass: 'java/io/InputStreamReader'
R9 =0x00000000ff575c80 is an oop: java.io.BufferedReader 
{0x00000000ff575c80} - klass: 'java/io/BufferedReader'
R10=0x00007f0c80f7a008 is an unknown value
R11=0x0000000000002000 is an unknown value
R12=0x0 is NULL
R13=0x00007f0c80f70548 is pointing into the stack for thread: 0x00007f0c7802bf10
R14=0x00007f0c80f705f8 is pointing into the stack for thread: 0x00007f0c7802bf10
R15=0x00007f0c7802bf10 is a thread


Registers:
RAX=0x000000000000006b, RBX=0x000000000000006b, RCX=0x000000000000166e, RDX=0x0000000040100001
RSP=0x00007f0c80f70538, RBP=0x00007f0c80f70598, RSI=0x00000000ff579d48, RDI=0x00000000c04aee68
R8 =0x00000000ff5737b8, R9 =0x00000000ff575c80, R10=0x00007f0c80f7a008, R11=0x0000000000002000
R12=0x0000000000000000, R13=0x00007f0c80f70548, R14=0x00007f0c80f705f8, R15=0x00007f0c7802bf10
RIP=0x00007f0c67e0c932, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f0c80f70538)
0x00007f0c80f70538:   00007f0c60339900 00007f0c60339900
0x00007f0c80f70548:   00000000ff579d48 00007f0c80f70550
0x00007f0c80f70558:   00007f0c5c7036d4 00007f0c80f705f8
0x00007f0c80f70568:   00007f0c5c703d60 00007f0c5c710940 

Instructions: (pc=0x00007f0c67e0c932)
0x00007f0c67e0c832:   c0 f0 4d 0f b1 7b 3e 48 c7 03 03 00 00 00 0f 85
0x00007f0c67e0c842:   06 02 00 00 41 8b 69 28 85 ed 0f 84 42 02 00 00
0x00007f0c67e0c852:   45 8b 59 14 45 8b 51 18 45 3b d3 0f 8d d3 00 00
0x00007f0c67e0c862:   00 41 0f b6 69 24 85 ed 0f 85 44 02 00 00 41 8b
0x00007f0c67e0c872:   49 18 41 8b 69 2c 44 8b d9 41 ff c3 45 89 59 18
0x00007f0c67e0c882:   44 8b 5d 0c 41 3b cb 0f 83 8d 01 00 00 0f b7 5c
0x00007f0c67e0c892:   4d 10 48 8d 44 24 30 48 81 38 00 00 00 00 0f 84
0x00007f0c67e0c8a2:   78 00 00 00 4d 8b 10 49 f7 c2 02 00 00 00 74 64
0x00007f0c67e0c8b2:   48 33 c0 49 0b 82 86 00 00 00 75 60 49 8b 82 96
0x00007f0c67e0c8c2:   00 00 00 49 0b 82 8e 00 00 00 75 0a 49 c7 42 3e
0x00007f0c67e0c8d2:   00 00 00 00 eb 46 49 81 ba 9e 00 00 00 00 00 00
0x00007f0c67e0c8e2:   00 74 25 48 33 c0 49 c7 42 3e 00 00 00 00 f0 83
0x00007f0c67e0c8f2:   04 24 00 49 81 ba 9e 00 00 00 00 00 00 00 75 0d
0x00007f0c67e0c902:   f0 4d 0f b1 7a 3e 75 05 83 c8 01 eb 0f a9 00 00
0x00007f0c67e0c912:   00 00 eb 08 4c 8b 10 f0 4d 0f b1 10 0f 85 53 01
0x00007f0c67e0c922:   00 00 8b c3 48 83 c4 40 5d 4d 8b 97 10 01 00 00
0x00007f0c67e0c932:   41 85 02 c3 49 8b f1 4c 89 4c 24 18 4c 89 44 24
0x00007f0c67e0c942:   08 4c 89 44 24 20 66 66 90 e8 d0 ab f1 f8 4c 8b
0x00007f0c67e0c952:   4c 24 18 45 8b 51 14 45 8b 41 18 45 3b c2 0f 8c
0x00007f0c67e0c962:   9f 00 00 00 48 8d 44 24 30 4c 8b 5c 24 08 48 81
0x00007f0c67e0c972:   38 00 00 00 00 0f 84 78 00 00 00 4d 8b 13 49 f7
0x00007f0c67e0c982:   c2 02 00 00 00 74 64 48 33 c0 49 0b 82 86 00 00
0x00007f0c67e0c992:   00 75 60 49 8b 82 96 00 00 00 49 0b 82 8e 00 00
0x00007f0c67e0c9a2:   00 75 0a 49 c7 42 3e 00 00 00 00 eb 46 49 81 ba
0x00007f0c67e0c9b2:   9e 00 00 00 00 00 00 00 74 25 48 33 c0 49 c7 42
0x00007f0c67e0c9c2:   3e 00 00 00 00 f0 83 04 24 00 49 81 ba 9e 00 00
0x00007f0c67e0c9d2:   00 00 00 00 00 75 0d f0 4d 0f b1 7a 3e 75 05 83
0x00007f0c67e0c9e2:   c8 01 eb 0f a9 00 00 00 00 eb 08 4c 8b 10 f0 4d
0x00007f0c67e0c9f2:   0f b1 13 0f 85 d9 00 00 00 bb ff ff ff ff e9 1f
0x00007f0c67e0ca02:   ff ff ff 4c 8b 44 24 08 e9 54 fe ff ff be f6 ff
0x00007f0c67e0ca12:   ff ff 49 8b e9 e8 64 16 57 f8 be e4 ff ff ff 48
0x00007f0c67e0ca22:   89 44 24 f8 8b 44 24 10 89 04 24 48 8b 44 24 f8
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.