Elasticsearch JVM Crashing occasionally

(Carsten König) #1

Hi, sometimes our elasticsearch just stopps working. It is no longer available via curl and the java processes are no longer running, but sadly the systemd daemon doesn't notice that - so it doesn't automatically restart.


$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.2 LTS
Release:	16.04
Codename:	xenial


$ curl localhost:9200
"name" : "xcRongv",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "_vh8wSYVQt2tNPlzQJYjVA",
"version" : {
  "number" : "5.3.2",
  "build_hash" : "3068195",
  "build_date" : "2017-04-24T16:15:59.481Z",
  "build_snapshot" : false,
  "lucene_version" : "6.4.2"
"tagline" : "You Know, for Search"


$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

The error is similiar to the other errors mentioned in Elastic search fails after some time . Syslog only says:

systemd[1]: elasticsearch.service: Main process exited, code=killed, status=6/ABRT

And in /tmp/hs_err_pid1110.log

# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0x00007f931557e371, pid=1110, tid=0x00007f93001d9700
# JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x624371]
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
... followed by a long dump.

complete error: https://pastebin.com/G3Ct0jY9

I have no idea why this happens, but it happens on multiple machines a few times a month. Any help would be appreciated!

(Mark Walkom) #2

What version of ES?

(Carsten König) #3

I included every version info in the original post. ES is 5.3.2.

(Jason Tedor) #4

This is a segfault in a GC thread (from the core dump), this is indicative of a JVM bug. I would recommend upgrading to the latest JVM (8u131) and crossing your fingers. If that does not work, you'll have to open a bug report with Canonical.

(Carsten König) #5

Thanks for the answers. I updated to the latest version of JVM, even if i am pretty sure that this won't fix it. Will report back when it happens again :slight_smile:

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.