Elastic search fails after some time

Elastic Stack Elasticsearch
1

Hello,

We have elasticsearch installed and configured to work with ActiveCollab on CentOS7 64 bit with 8GB RAM. ElasticSearch starts successfully but fails after some time. The status message shows:

elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Thu 2016-01-21 11:05:55 GST; 1h 17min ago
Docs: http://www.elastic.co
Main PID: 206246 (code=killed, signal=ABRT)
Started Elasticsearch.
Starting Elasticsearch...
elasticsearch.service: main process exited, code=killed, status=6/ABRT
Unit elasticsearch.service entered failed state.
elasticsearch.service failed.

Couldn't find the reason/fix yet. Please help

2

Something is killing it by the looks of things.
Try checking your primary system logs.

3

Didn't find anything specific from the logs. Is there any specific things i can check to find this?

4

Which logs did you check?

5

/var/log/elasticsearch/elasticsearch

6

And what about the system level logs, as I mentioned.

7

Checked messages, if thats what you are asking about, but that gives the same as elasticsearch status

systemd: elasticsearch.service: main process exited, code=killed, status=6/ABRT
systemd: Unit elasticsearch.service entered failed state.
systemd: elasticsearch.service failed.

8

Just started journalctl -u elasticsearch.service -f, will see if it shows anything more when this fails. As per one post, i just changed the StandardOutput=null to StandardOutput=journal

9

I got some detailed error which relates to Java.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f3d5ec2a3cc, pid=971889, tid=139901534000896

JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)

Problematic frame:

V [libjvm.so+0x93c3cc] ParRootScanWithoutBarrierClosure::do_oop(oopDesc**)+0x6c

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

If you would like to submit a bug report, please visit:

http://bugreport.java.com/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x00007f3d5801a000): GCTaskThread [stack: 0x00007f3d5d2cd000,0x00007f3d5d3ce000] [id=971897]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000002400288

10

Is there anything else that i could check? Increased the heap size to half of the server RAM but it still failed!

11

Oh dear, looks like you found a bug in the JVM. :tired_face:

I skimmed the jvm bugtracker and didn't see a related bug, but I may have missed it. The easiest route is to upgrade your JVM to most recent (1_8_u71) and hope the bug is fixed.

Are you using G1GC by chance? The segfault was thrown from a GC thread, and it's known that G1GC is not stable yet and throws segfaults on occasion :slight_smile:

1 Like
12

Thanks for the help. I am not very sure on how to upgrade to the recent java version. Followed the documentation from https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-elasticsearch-on-centos-7

13

I installed ES on a separate server to avoid resources being shared with anything else. Now i need to give access to a remote server to ES, I couldn't figure out how to do that. Ports are opened but can't connect. Please help

14

Did you change network.host to bind to anything but localhost?

15

Below are my settings now:

network.bind_host: localhost
network.publish_host: localhost
network.host: localhost

16

Change network.host and remove the 2 others.

1 Like
17

Thanks, change the network.host to the Public IP of the server?

18

Public or private. But to an IP your other server can see.

1 Like
19

Thank you so much, that worked! :slight_smile:

20

I am able to access the elastic search on public IP on port 9200, but unable to connect ActiveCollab to ES. Still checking this now