Elasticsearch JVM Crashing occasionally

[cars10 (Carsten König)]

Hi, sometimes our elasticsearch just stopps working. It is no longer available via curl and the java processes are no longer running, but sadly the systemd daemon doesn't notice that - so it doesn't automatically restart.

System:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.2 LTS
Release:	16.04
Codename:	xenial

Elasticsearch:

$ curl localhost:9200
{
"name" : "xcRongv",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "_vh8wSYVQt2tNPlzQJYjVA",
"version" : {
  "number" : "5.3.2",
  "build_hash" : "3068195",
  "build_date" : "2017-04-24T16:15:59.481Z",
  "build_snapshot" : false,
  "lucene_version" : "6.4.2"
},
"tagline" : "You Know, for Search"
}

Java:

$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

The error is similiar to the other errors mentioned in Elastic search fails after some time . Syslog only says:

systemd[1]: elasticsearch.service: Main process exited, code=killed, status=6/ABRT

And in /tmp/hs_err_pid1110.log

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f931557e371, pid=1110, tid=0x00007f93001d9700
#
# JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x624371]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
... followed by a long dump.

complete error: https://pastebin.com/G3Ct0jY9

I have no idea why this happens, but it happens on multiple machines a few times a month. Any help would be appreciated!

[warkolm (Mark Walkom)]

What version of ES?

[cars10 (Carsten König)]

I included every version info in the original post. ES is 5.3.2.

[jasontedor]

This is a segfault in a GC thread (from the core dump), this is indicative of a JVM bug. I would recommend upgrading to the latest JVM (8u131) and crossing your fingers. If that does not work, you'll have to open a bug report with Canonical.

[cars10 (Carsten König)]

Thanks for the answers. I updated to the latest version of JVM, even if i am pretty sure that this won't fix it. Will report back when it happens again