Elasticsearch 8.12.2 Startup Failure: SIGILL Crash

Elastic Stack Elasticsearch
1

Hello everyone, thank you so much for taking the time to read this. I have a small problem with my Elastic server:

My Elasticsearch 8.12.2 was running without issues for several months on multiple nodes in a Red Hat Enterprise Linux 9.3 environment. Recently,
the service started failing on startup with a SIGILL (Illegal Instruction) error.

Error message starting elasticsearch service:

nov 20 10:45:36 myserver systemd[1]: Starting Node 6 - Elastic-Search Cluster MyServer PRO...
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver elasticsearch[2996623]: # A fatal error has been detected by the Java Runtime Environment:
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver elasticsearch[2996623]: # SIGILL (0x4) at pc=0x00007f510805b62c, pid=2996623, tid=2996641
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver elasticsearch[2996623]: # JRE version: OpenJDK Runtime Environment (21.0.2+13) (build 21.0.2+13-58)
nov 20 10:45:36 myserver elasticsearch[2996623]: # Java VM: OpenJDK 64-Bit Server VM (21.0.2+13-58, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)
nov 20 10:45:36 myserver elasticsearch[2996623]: # Problematic frame:
nov 20 10:45:36 myserver elasticsearch[2996623]: # V [libjvm.so+0xc5b62c] MethodData::initialize()+0x18c
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver elasticsearch[2996623]: # Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /opt/elk/elastic/core.2996623)
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver elasticsearch[2996623]: # An error report file with more information is saved as:
nov 20 10:45:36 myserver elasticsearch[2996623]: # /opt/elk/elastic/hs_err_pid2996623.log
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver elasticsearch[2996623]: # Compiler replay data is saved as:
nov 20 10:45:36 myserver elasticsearch[2996623]: # /opt/elk/elastic/replay_pid2996623.log
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver elasticsearch[2996623]: # If you would like to submit a bug report, please visit:
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver elasticsearch[2996623]: #
nov 20 10:45:36 myserver systemd-coredump[2996645]: [🡕] Process 2996623 (java) of user 1000 dumped core.
nov 20 10:45:36 myserver systemd[1]: elk-elastic.service: Control process exited, code=dumped, status=6/ABRT
nov 20 10:45:36 myserver systemd[1]: elk-elastic.service: Failed with result 'core-dump'.
nov 20 10:45:36 myserver systemd[1]: Failed to start Node 6 - MyServer - PRO.
nov 20 10:45:36 myserver systemd[1]: elk-elastic.service: Scheduled restart job, restart counter is at 1.
nov 20 10:45:36 myserver systemd[1]: Stopped Node 6 - MyServer - PRO.

In the hs_err_pid2996623.log file appears:

A fatal error has been detected by the Java Runtime Environment:

SIGILL (0x4) at pc=0x00007f00a6e5b62c, pid=2996720, tid=2996741

JRE version: OpenJDK Runtime Environment (21.0.2+13) (build 21.0.2+13-58)

Java VM: OpenJDK 64-Bit Server VM (21.0.2+13-58, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)

Problematic frame:

V  [libjvm.so+0xc5b62c]  MethodData::initialize()+0x18c

Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /opt/elk/elastic/core.2996720)



---------------  S U M M A R Y ------------

Command Line: -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/opt/elk/elastic/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/opt/elk/elastic -Des.path.conf=/opt/elk/elastic/config -Des.distribution.type=tar org.elasticsearch.launcher.CliToolLauncher -d -p /opt/elk/cluster-elk/data-elastic/elastic.pid

Host: Intel(R) Xeon(R) Gold 6258R CPU @ 2.70GHz, 16 cores, 31G, Red Hat Enterprise Linux release 9.3 (Plow)
Time: Thu Nov 20 10:45:37 2025 WET elapsed time: 0.049523 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x00007f002002cb80):  JavaThread "C1 CompilerThread1" daemon [_thread_in_vm, id=2996741, stack(0x00007f00a4f27000,0x00007f00a5027000) (1024K)]

Current CompileTask:
C1:     49   23   !   3       java.util.zip.ZipFile$Source::checkAndAddEntry (295 bytes)

Stack: [0x00007f00a4f27000,0x00007f00a5027000],  sp=0x00007f00a5024720,  free space=1013k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

…
siginfo: si_signo: 4 (SIGILL), si_code: 2 (ILL_ILLOPN), si_addr: 0x00007f00a6e5b62c

Registers:
RAX=0x0000000000000001, RBX=0x00007f003c017408, RCX=0x0000000000000003, RDX=0x00007f00a7614680
RSP=0x00007f00a5024720, RBP=0x00007f00a50247f0, RSI=0x00007f00a5024790, RDI=0x00007f003c017408
R8 =0x00007f0047319008, R9 =0x0000000000000004, R10=0x0000000000000009, R11=0x0000000000000000
R12=0x0000000000000000, R13=0x0000000000000000, R14=0x00007f00a5024790, R15=0x00000000000000bb
RIP=0x00007f00a6e5b62c, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000000
TRAPNO=0x0000000000000006

Register to memory mapping:

RAX=0x0000000000000001 is an unknown value
RBX=0x00007f003c017408 is pointing into metadata
RCX=0x0000000000000003 is an unknown value
RDX=0x00007f00a7614680: <offset 0x0000000001414680> in /opt/elk/elastic/jdk/lib/server/libjvm.so at 0x00007f00a6200000
RSP=0x00007f00a5024720 is pointing into the stack for thread: 0x00007f002002cb80
RBP=0x00007f00a50247f0 is pointing into the stack for thread: 0x00007f002002cb80
RSI=0x00007f00a5024790 is pointing into the stack for thread: 0x00007f002002cb80
RDI=0x00007f003c017408 is pointing into metadata
R8 ={method} {0x00007f0047319008} 'zerror' '(Ljava/lang/String;)V' in 'java/util/zip/ZipFile$Source'
R9 =0x0000000000000004 is an unknown value
R10=0x0000000000000009 is an unknown value
...

A summary of my server is as follows:

  • Elasticsearch version: 8.12.2
  • Kibana version: 8.12.2
  • Java version: OpenJDK 21.0.2+13-58 (bundled)
  • OS: Red Hat Enterprise Linux 9.3
  • CPU: Intel Xeon Gold 6258R @ 2.70GHz

If we start with an OpenJDK 64-Bit Server VM (Red Hat build 21.0.9+10-LTS), it boots up without any problems. Is it possible to use this one, or is there a way to fix this issue?

Best regards and thank you in advance

Pabedi

2

welcome to the forum @Pabendi

Thanks for providing the info supplied.

Just to be clear, it (elasticsearch + kibana 8.12.2) worked for ages with the bundled JDK on the same RHEL9.3 server, then suddenly stopped working. And if you try the different (slightly newer) JVM it works again?

In principle you can use loads of JVMs, even one you built yourself. So, if above is the case, you should be fine. Many people use custom JVMs for different reasons.

If very curious you can check if some RHEL patch has created an issue with the specific JVM.

Someone will point out that 8.12.2 is now 18months+ old and you may wish to consider upgrading to a newer release, which will also bring a newer JVM.

1 Like
3

Yes Kevin, we have tested with Red Hat OpenJDK 21.0.9+10-LTS and it works. We hope this resolves the issue and we have no further problems.

If you or anyone else has additional info or perspectives, please feel free to share.

Thank you so much for your help!

Pabendi

4

The support matrix is here

For 8.12 there’s just openjdk 17 and 21 with green ticks, so I recommend to stick to 21.

There’s also the changelogs for each openJDK release, but you would need to be fortunate to find a specific bug and fix, if indeed it is a specific fix, that has resolved your issue.

1 Like