Elasticsearch starts then stops

Elastic Stack Elasticsearch
1

Hello,
I have installed elasticsearch (elasticsearch-2.3.3-1) and kibana (kibana-4.5.1) on one of our VMs ( with RHEL 7.5). Kibana starts without any issue. However, elasticsearch starts and then dies. Here's what I see when I start elasticsearch:

$ sudo systemctl start elasticsearch
$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2018-07-07 05:46:32 UTC; 8s ago
Docs: http://www.elastic.co
Process: 24261 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 24263 (java)
CGroup: /system.slice/elasticsearch.service
└─24263 [java]
‣ 24263 [java]

Jul 07 05:46:31 lma-portal systemd[1]: Starting Elasticsearch...
Jul 07 05:46:32 lma-portal systemd[1]: Started Elasticsearch.
$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Sat 2018-07-07 05:46:45 UTC; 6s ago
Docs: http://www.elastic.co
Process: 24263 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=killed, signal=KILL)
Process: 24261 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 24263 (code=killed, signal=KILL)

Jul 07 05:46:31 lma-portal systemd[1]: Starting Elasticsearch...
Jul 07 05:46:32 lma-portal systemd[1]: Started Elasticsearch.
Jul 07 05:46:45 lma-portal systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
Jul 07 05:46:45 lma-portal systemd[1]: Unit elasticsearch.service entered failed state.
Jul 07 05:46:45 lma-portal systemd[1]: elasticsearch.service failed.

/var/log/messages
...
Jul 7 05:47:57 lma audispd: node=lma-portal type=USER_CMD msg=audit(1530942477.707:31048): pid=24375 uid=325400240 auid=325400240 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd="/opt/kibana/config" cmd=7461696C202D66202F7661722F6C6F672F6D65737361676573 terminal=pts/0 res=success'
Jul 7 05:47:57 lma audispd: node=lma-portal type=CRED_REFR msg=audit(1530942477.708:31049): pid=24375 uid=0 auid=325400240 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_localuser,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
Jul 7 05:47:57 lma audispd: node=lma-portal type=USER_START msg=audit(1530942477.708:31050): pid=24375 uid=0 auid=325400240 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:kibana","info"],"pid":24363,"name":"plugin:kibana","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:elasticsearch","info"],"pid":24363,"name":"plugin:elasticsearch","state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:kbn_vislib_vis_types","info"],"pid":24363,"name":"plugin:kbn_vislib_vis_types","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["error","elasticsearch"],"pid":24363,"message":"Request error, retrying -- connect ECONNREFUSED 127.0.0.1:9200"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:markdown_vis","info"],"pid":24363,"name":"plugin:markdown_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:metric_vis","info"],"pid":24363,"name":"plugin:metric_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["warning","elasticsearch"],"pid":24363,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["warning","elasticsearch"],"pid":24363,"message":"No living connections"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:elasticsearch","error"],"pid":24363,"name":"plugin:elasticsearch","state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://localhost:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:spyModes","info"],"pid":24363,"name":"plugin:spyModes","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:statusPage","info"],"pid":24363,"name":"plugin:statusPage","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["status","plugin:table_vis","info"],"pid":24363,"name":"plugin:table_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 7 05:47:58 lma kibana: {"type":"log","@timestamp":"2018-07-07T05:47:58+00:00","tags":["listening","info"],"pid":24363,"message":"Server running at http://0.0.0.0:5601"}
...

I have checked and double checked my configuration files and can't find anything.

Appreciate any help on this.

Thanks.
jb1462

2

Is there a specific reason you are running such an old version?

Have you looked at /var/log/elasticsearch/?

3

It's what's installed in all our other environments.

There's a few log files in /etc/log/elasticsearch but they're all empty.

$ ls -l /var/log/elasticsearch/
total 0
-rw-r--r--. 1 elasticsearch elasticsearch 0 Jul 6 05:55 lma-elk_index_indexing_slowlog.log
-rw-r--r--. 1 elasticsearch elasticsearch 0 Jul 6 05:55 lma-elk_index_search_slowlog.log
-rw-r--r--. 1 elasticsearch elasticsearch 0 Jul 6 05:55 lma-elk.log

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.