Elasticsearch JVM Crashing occasionally

Hi, sometimes our elasticsearch just stopps working. It is no longer available via curl and the java processes are no longer running, but sadly the systemd daemon doesn't notice that - so it doesn't automatically restart.


$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.2 LTS
Release:	16.04
Codename:	xenial


$ curl localhost:9200
"name" : "xcRongv",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "_vh8wSYVQt2tNPlzQJYjVA",
"version" : {
  "number" : "5.3.2",
  "build_hash" : "3068195",
  "build_date" : "2017-04-24T16:15:59.481Z",
  "build_snapshot" : false,
  "lucene_version" : "6.4.2"
"tagline" : "You Know, for Search"


$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

The error is similiar to the other errors mentioned in Elastic search fails after some time . Syslog only says:

systemd[1]: elasticsearch.service: Main process exited, code=killed, status=6/ABRT

And in /tmp/hs_err_pid1110.log

# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0x00007f931557e371, pid=1110, tid=0x00007f93001d9700
# JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x624371]
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
... followed by a long dump.

complete error: https://pastebin.com/G3Ct0jY9

I have no idea why this happens, but it happens on multiple machines a few times a month. Any help would be appreciated!

What version of ES?

I included every version info in the original post. ES is 5.3.2.

This is a segfault in a GC thread (from the core dump), this is indicative of a JVM bug. I would recommend upgrading to the latest JVM (8u131) and crossing your fingers. If that does not work, you'll have to open a bug report with Canonical.

1 Like

Thanks for the answers. I updated to the latest version of JVM, even if i am pretty sure that this won't fix it. Will report back when it happens again :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.