Elastcisearch process gets killed automatically

Elastic Stack Elasticsearch
1

Hello,

I am new to ELK, recently configured AWS-Linux box with elastcisearch and kibana.

Process for both kibana and elastcisearch started successfully and able to get the logs and corresponding results.

But after sometime elastciseach process gets killed automatically, ie i have to start the process again to get active. Could you please suggest any help on this?

elastciseach logs:-

dashboard]$ ./elasticsearch-2.3.3/bin/elasticsearch
[2016-07-16 21:31:26,738][INFO ][node ] [Centurion] version[2.3.3], pid[23133], build[218bdf1/2016-05-17T15:40:04Z]
[2016-07-16 21:31:26,739][INFO ][node ] [Centurion] initializing ...
[2016-07-16 21:31:28,133][INFO ][plugins ] [Centurion] modules [lang-groovy, reindex, lang-expression], plugins [], sites []
[2016-07-16 21:31:28,179][INFO ][env ] [Centurion] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [5.9gb], net total_space [7.7gb], spins? [no], types [ext4]
[2016-07-16 21:31:28,179][INFO ][env ] [Centurion] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-07-16 21:31:28,180][WARN ][env ] [Centurion] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-07-16 21:31:32,469][INFO ][node ] [Centurion] initialized
[2016-07-16 21:31:32,469][INFO ][node ] [Centurion] starting ...
[2016-07-16 21:31:32,596][INFO ][transport ] [Centurion] publish_address {xxx.xxx.xxx.xxx:9300}, bound_addresses {xxx.xxx.xxx.xxx:9300}
[2016-07-16 21:31:32,605][INFO ][discovery ] [Centurion] elasticsearch/f-BQjtvWQCaunaOLu8OJkA
[2016-07-16 21:31:35,805][INFO ][cluster.service ] [Centurion] new_master {Centurion}{f-BQjtvWQCaunaOLu8OJkA}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-07-16 21:31:35,837][INFO ][http ] [Centurion] publish_address {xxx.xxx.xxx.xxx:9200}, bound_addresses {xxx.xxx.xxx.xxx:9200}
[2016-07-16 21:31:35,838][INFO ][node ] [Centurion] started
[2016-07-16 21:31:35,962][INFO ][gateway ] [Centurion] recovered [2] indices into cluster_state
[2016-07-16 21:31:37,238][INFO ][cluster.routing.allocation] [Centurion] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2016.07.07][4]] ...]).

2

Please format your logs with </> icon.

Do you start Elasticsearch with bin/elasticsearch?
So anytime you log out, the process is stopped because it runs in foreground.

You should run Elasticsearch as a service instead.
Use Deb or rpm packages instead.

1 Like
3

Thank you David for the response!

I am using command -- "bin/elasticsearch & " for running in background.

Also could you please elaborate "format your logs with </> icon" ?

I am using logs format as below

Log file:-

[LAYER]GSDIM, [TYPE]REQ, [METHOD]SESSION, [ACTION]START, [latitude]47.60621,[longitude]-122.33207
[LAYER]GSDIM, [TYPE]REP, [TIME]1, [METHOD]SESSION, [ACTION]START, [RESPONSE]Session Started,[latitude]47.60621,[longitude]-122.33207

4

I meant that instead of posting here:

dashboard]$ ./elasticsearch-2.3.3/bin/elasticsearch
[2016-07-16 21:31:26,738][INFO ][node ] [Centurion] version[2.3.3], pid[23133], build[218bdf1/2016-05-17T15:40:04Z]
[2016-07-16 21:31:26,739][INFO ][node ] [Centurion] initializing ...

Do:

dashboard]$ ./elasticsearch-2.3.3/bin/elasticsearch
[2016-07-16 21:31:26,738][INFO ][node ] [Centurion] version[2.3.3], pid[23133], build[218bdf1/2016-05-17T15:40:04Z]
[2016-07-16 21:31:26,739][INFO ][node ] [Centurion] initializing ...
1 Like
5

Even though this runs in the background it will still get killed when you logout. I definitely agree with the recommendation to use existing RPM or DEBs and to run this as a service.

However, if you really don't want to run it as a service, you can detach the background process from your session if you want it to continue when you logout:

# bin/elasticsearch >& /dev/null &
# disown %1
1 Like
6

Hello,

Just now changed the process to run as a service using RPM, but still have the issue of getting automatically stopped/killed.

Couple of other observations:-

  1. If I am starting only elastcisearch service then elastcisearch service will not gets killed/stopped.

sudo service elasticsearch start

  1. If i am starting elasctisearch and kibana in same server then elastcisearch service will gets killed/stopped after some time.

sudo service elasticsearch start
sudo service kibana start

Any logs i can check here for this issue? Any help appreciated!

7

dmesg and /var/log/messages might show any system errors like out of memory conditions.

1 Like