Elastic search fails after some time

Elastic Stack Elasticsearch
21

Sorted that too, it was a local firewall issue. Thank all. Now need to monitor ES for some time to make sure it doesnt fail!

22

I am having this same problem. Was a solution found for this?

Elasticsearch crashes after a short amount of time is restarts. Nothing in the elasticsearch log. The error is similar to OP's

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fb11a9f67c4, pid=976388, tid=140396317169408

JRE version: OpenJDK Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)

Java VM: OpenJDK 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops)

Problematic frame:

V [libjvm.so+0x5387c4]

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

If you would like to submit a bug report, please visit:

http://bugreport.java.com/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x00007fafd400a800): JavaThread "elasticsearch[Anti-Cap][management][T#1]" daemon [_thread_in_vm, id=976482, stack(0x00007fb0908cd000,0x00007fb0909ce000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000

23

What version of ES are you on? If it's 1.x, it may be from our usage of Sigar (used to collect host stats/metrics). Iirc there have been problems with Sigar throwing segfaults in the past.

We removed Sigar in 2.x. So if you're on 2.0+, the problem is likely just a bug in the JVM and I'm not really sure we can help much. I'm not aware of any outstanding bugs that affect ES on u71, but perhaps there is a new one you've run into. I'd file a bug report at Oracle/OpenJDK with as much reproduction details as you can.

Are you using G1 GC? If yes, can you switch back to CMS and see if the problem continues? G1 is still ... dicey. For example, bugs like this still exist: Loading... , so who knows what kind of problems it causes :slight_smile:

24

Upgrading Java to 1.8 (from 1.7) and setting ulimit-c unlimited seems to have fixed the problem.

FYI, I am running ES 1.7.5

25

I experience the same issue. Sometimes elasticsearch just stopps, but sadly the daemon does not - so it does not automatically restart.

System:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.2 LTS
Release:	16.04
Codename:	xenial

Elasticsearch:

$ curl localhost:9200
{
"name" : "xcRongv",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "_vh8wSYVQt2tNPlzQJYjVA",
"version" : {
  "number" : "5.3.2",
  "build_hash" : "3068195",
  "build_date" : "2017-04-24T16:15:59.481Z",
  "build_snapshot" : false,
  "lucene_version" : "6.4.2"
},
"tagline" : "You Know, for Search"
}

Java:

$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

The error is similiar to the other errors mentioned here. Syslog only mentions:

systemd[1]: elasticsearch.service: Main process exited, code=killed, status=6/ABRT

And in /tmp/hs_err_pid1110.log

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f931557e371, pid=1110, tid=0x00007f93001d9700
#
# JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x624371]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
... followed by a long dump.

complete error: https://pastebin.com/G3Ct0jY9

26

This thread is over a year old, please start a new one :slight_smile: