Elastic search fails after some time


(Ambarishrh) #1

Hello,

We have elasticsearch installed and configured to work with ActiveCollab on CentOS7 64 bit with 8GB RAM. ElasticSearch starts successfully but fails after some time. The status message shows:

elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Thu 2016-01-21 11:05:55 GST; 1h 17min ago
Docs: http://www.elastic.co
Main PID: 206246 (code=killed, signal=ABRT)
Started Elasticsearch.
Starting Elasticsearch...
elasticsearch.service: main process exited, code=killed, status=6/ABRT
Unit elasticsearch.service entered failed state.
elasticsearch.service failed.

Couldn't find the reason/fix yet. Please help


1.7.6 dies after a few hours - Disable Sigar how?
(Mark Walkom) #2

Something is killing it by the looks of things.
Try checking your primary system logs.


(Ambarishrh) #3

Didn't find anything specific from the logs. Is there any specific things i can check to find this?


(Mark Walkom) #4

Which logs did you check?


(Ambarishrh) #5

/var/log/elasticsearch/elasticsearch


(Mark Walkom) #6

And what about the system level logs, as I mentioned.


(Ambarishrh) #7

Checked messages, if thats what you are asking about, but that gives the same as elasticsearch status

systemd: elasticsearch.service: main process exited, code=killed, status=6/ABRT
systemd: Unit elasticsearch.service entered failed state.
systemd: elasticsearch.service failed.


(Ambarishrh) #8

Just started journalctl -u elasticsearch.service -f, will see if it shows anything more when this fails. As per one post, i just changed the StandardOutput=null to StandardOutput=journal


(Ambarishrh) #9

I got some detailed error which relates to Java.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f3d5ec2a3cc, pid=971889, tid=139901534000896

JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)

Problematic frame:

V [libjvm.so+0x93c3cc] ParRootScanWithoutBarrierClosure::do_oop(oopDesc**)+0x6c

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

If you would like to submit a bug report, please visit:

http://bugreport.java.com/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x00007f3d5801a000): GCTaskThread [stack: 0x00007f3d5d2cd000,0x00007f3d5d3ce000] [id=971897]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000002400288


(Ambarishrh) #10

Is there anything else that i could check? Increased the heap size to half of the server RAM but it still failed!


(Zachary Tong) #11

Oh dear, looks like you found a bug in the JVM. :tired_face:

I skimmed the jvm bugtracker and didn't see a related bug, but I may have missed it. The easiest route is to upgrade your JVM to most recent (1_8_u71) and hope the bug is fixed.

Are you using G1GC by chance? The segfault was thrown from a GC thread, and it's known that G1GC is not stable yet and throws segfaults on occasion :slight_smile:


(Ambarishrh) #12

Thanks for the help. I am not very sure on how to upgrade to the recent java version. Followed the documentation from https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-elasticsearch-on-centos-7


(Ambarishrh) #13

I installed ES on a separate server to avoid resources being shared with anything else. Now i need to give access to a remote server to ES, I couldn't figure out how to do that. Ports are opened but can't connect. Please help


(David Pilato) #14

Did you change network.host to bind to anything but localhost?


(Ambarishrh) #15

Below are my settings now:

network.bind_host: localhost
network.publish_host: localhost
network.host: localhost


(David Pilato) #16

Change network.host and remove the 2 others.


(Ambarishrh) #17

Thanks, change the network.host to the Public IP of the server?


(David Pilato) #18

Public or private. But to an IP your other server can see.


(Ambarishrh) #19

Thank you so much, that worked! :slight_smile:


(Ambarishrh) #20

I am able to access the elastic search on public IP on port 9200, but unable to connect ActiveCollab to ES. Still checking this now