Elasticsearch won't upstart

Hi!

I have been successfully running ELK for a day :slight_smile:

However, after a needed server restart I cannot upstart Elasticsearch from systemctl.

If I run it manually it works.

My set up:

centos7
elasticsearch-2.2.1-1.noarch

Running from systemctl gives:

` [root@localhost share]# sudo /bin/systemctl status elasticsearch.service
ā— elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2016-03-28 01:26:45 BST; 21s ago
Docs: http://www.elastic.co
Process: 3936 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=127)
Process: 3934 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 3936 (code=exited, status=127)

Mar 28 01:26:45 localhost.localdomain systemd[1]: Starting Elasticsearch...
Mar 28 01:26:45 localhost.localdomain systemd[1]: Started Elasticsearch.
Mar 28 01:26:45 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=127/n/a
Mar 28 01:26:45 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Mar 28 01:26:45 localhost.localdomain systemd[1]: elasticsearch.service failed.`

All files/folders seem to be in place.

Here is the .service config:

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Environment=ES_HOME=/usr/share/elasticsearch
Environment=CONF_DIR=/etc/elasticsearch
Environment=DATA_DIR=/var/lib/elasticsearch
Environment=LOG_DIR=/var/log/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/sysconfig/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec

ExecStart=/usr/share/elasticsearch/bin/elasticsearch
-Des.pidfile=${PID_DIR}/elasticsearch.pid
-Des.default.path.home=${ES_HOME}
-Des.default.path.logs=${LOG_DIR}
-Des.default.path.data=${DATA_DIR}
-Des.default.path.conf=${CONF_DIR}
StandardOutput=journal
StandardError=inherit

Specifies the maximum file descriptor number that can be opened by this process

LimitNOFILE=65535

Specifies the maximum number of bytes of memory that may be locked into RAM

Set to "infinity" if you use the 'bootstrap.mlockall: true' option

in elasticsearch.yml and 'MAX_LOCKED_MEMORY=unlimited' in /etc/sysconfig/elasticsearch

#LimitMEMLOCK=infinity

Disable timeout logic and wait until process is stopped

TimeoutStopSec=0

SIGTERM signal is used to stop the Java process

KillSignal=SIGTERM

Java process is never killed

SendSIGKILL=no

When a JVM receives a SIGTERM signal it exits with code 143

SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

Built for Distribution: RPM-2.2.1 (rpm)`

elasticsearch user is owner of all the above mentioned folders.. I'm not sure where to go.

Any help will be much appreciated!

What do your ES logs show?

Actually I'm not able to find any related logs when is start with systemctl. Based on the config file I expect to find them:

/var/log/elasticsearch/

But nothing is going there anymore. When I start it up manually the logs go into:

/usr/share/elasticsearch/logs/elastic.log

I believe I'm ending in similar situation on CentOS Linux release 7.2.1511 (Core) with 5.0.0_alpha4-1.

[root@lab elasticsearch]# service elasticsearch status
ā— elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-07-06 18:16:35 CEST; 7min ago
     Docs: http://www.elastic.co
  Process: 45928 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
  Process: 45926 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 45928 (code=exited, status=1/FAILURE)

Jul 06 18:16:35 lab systemd[1]: Starting Elasticsearch...
Jul 06 18:16:35 lab systemd[1]: Started Elasticsearch.
Jul 06 18:16:35 lab systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 06 18:16:35 lab systemd[1]: Unit elasticsearch.service entered failed state.
Jul 06 18:16:35 lab systemd[1]: elasticsearch.service failed.

Main problem is that there was no log produced, so I'm not sure how to debug it further.

I also ran into the same situation.
I was using ES 5.0 on CentOS 7.
I downgraded it to CentOS 6 and now it's working fine.

Get same issue here, I am restarting it and then it stucked there. There is no elasticsearch proces but I cannot start it successfully.

Hi , I have faced the same problem in RHEL where the elasticsearch won't generate logs .
Then The problem was due to an extra space in the elasticsearch.yml file in /etc/elasticsearch path.