Node crashes without logs randomly. StubRoutines::jbyte_disjoint_arraycopy

[imehl (Michael)]

One node of my cluster randomly crashes without any log in /var/log/elasticsearch and I don't know how to solve/debug this problem

Here is the hs_err_pid.log
https://pastebin.com/tKiGSUPh

There is enough free space (root, data paths, tmp - multiple disks are used) and memory.

java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

elasticsearch --version
Version: 6.2.2, Build: 10b1edd/2018-02-16T19:01:30.685723Z, JVM: 1.8.0_121

[spinscale (Alexander Reelsen)]

Hey,

so the error in the log is a SIGBUS error, which is pretty generic and can hint at a hardware or a software failure. The JVM you are using already is a bit older, maybe you can try to upgrade that one first and see if the problem persists?

What is the nature of those crashes? Doing the same operation or rather random? At the same time of the day? How about the frquency?

--Alex

[imehl (Michael)]

Thanks for your answer.

The cluster is running since a year (Upgrade to 6.2.2 one month ago; 10 nodes on bare metal, hot-warm architecture) without problems. There were 3 crashes on the same node (a warm node) in the last 7 days at different time and I also can't see any context to other actions. At the time of the crash there were no indexing/allocation/snapshot jobs running - only queries. So for me it is random.

I will check hardware and upgrade Java.

[imehl (Michael)]

Update: I upgraded Java to 1.8.0_161 - but had a crash again