I'm a beginner with elastic search, the same was put into production was with Kibana, search was working but due to some update its stop working and giving this error message while checking status.
========================================================================
service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2018-07-12 08:53:06 -03; 25min ago
Docs: http://www.elastic.co
Process: 2898 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2898 (code=exited, status=1/FAILURE)
Jul 12 08:53:06 a6-zwatcher1 systemd[1]: Started Elasticsearch.
Jul 12 08:53:06 a6-zwatcher1 systemd[1]: Starting Elasticsearch...
Jul 12 08:53:06 a6-zwatcher1 elasticsearch[2898]: which: no java in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
Jul 12 08:53:06 a6-zwatcher1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 12 08:53:06 a6-zwatcher1 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 12 08:53:06 a6-zwatcher1 systemd[1]: elasticsearch.service failed.
========================================================================
Log file:
tail -f elasticsearch.log
[2018-07-11T17:16:55,023][INFO ][o.e.c.r.a.DiskThresholdMonitor] [tef-elk-01] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-07-11T17:17:25,740][WARN ][o.e.c.r.a.DiskThresholdMonitor] [tef-elk-01] high disk watermark [90%] exceeded on [UuMB8U6dSGCJ3BUujtbFHQ][tef-elk-02][/export/elasticsearch/data/nodes/0] free: 22.6gb[9.2%], shards will be relocated away from this node
[2018-07-11T17:17:55,875][WARN ][o.e.c.r.a.DiskThresholdMonitor] [tef-elk-01] high disk watermark [90%] exceeded on [UuMB8U6dSGCJ3BUujtbFHQ][tef-elk-02][/export/elasticsearch/data/nodes/0] free: 22.6gb[9.2%], shards will be relocated away from this node
[2018-07-11T17:17:55,875][INFO ][o.e.c.r.a.DiskThresholdMonitor] [tef-elk-01] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-07-11T17:18:26,341][WARN ][o.e.c.r.a.DiskThresholdMonitor] [tef-elk-01] high disk watermark [90%] exceeded on [UuMB8U6dSGCJ3BUujtbFHQ][tef-elk-02][/export/elasticsearch/data/nodes/0] free: 22.6gb[9.2%], shards will be relocated away from this node
However the same is not with the full partition:
df -h /var/opt/UOLI/elasticsearch/data
Filesystem Size Used Avail Use% Mounted on
server-2c-trk1:/uolps_tefelasticsearch_0 247G 224G 23G 91% /var/opt/UOLI/elasticsearch/data