Hi
I'm having a single node running ES but it wont start suddently.
Logs showing:
SYSLOG: Jul 30 13:53:49 rg-app-prod-elk kibana[1495]: {"type":"log","@timestamp":"2018-07-30T13:53:49Z","tags":["warning","elasticsearch","admin"],"pid":1495,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 30 13:53:49 rg-app-prod-elk kibana[1495]: {"type":"log","@timestamp":"2018-07-30T13:53:49Z","tags":["warning","elasticsearch","admin"],"pid":1495,"message":"No living connections"}
ES node log:
[2018-07-04T23:19:42,399][WARN ][o.e.c.r.a.DiskThresholdMonitor] [PROD ELK STACK] high disk watermark [90%] exceeded on [LLArx98CRD-mXivQicPXTA][PROD ELK STACK][/var/lib/elasticsearch/PROD-elasticsearch/nodes/0] free: 2.8gb[9.7%], shards will be relocated away from this node
[2018-07-04T23:19:42,399][INFO ][o.e.c.r.a.DiskThresholdMonitor] [PROD ELK STACK] rerouting shards: [high disk watermark exceeded on one or more nodes]
ES status:
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-07-30 14:25:35 UTC; 27s ago
Docs: http://www.elastic.co
Process: 2636 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 2631 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 2636 (code=exited, status=1/FAILURE)
CPU: 2.569s
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: # ----------------------------------- Memory -----------------------------------
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: #
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: # Lock the memory on startup:
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: #
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: bootstrap.memory_lock: true
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: #
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: # Make sure that the ES_HEAP_SIZE
environment variable is set to about half the memory
Jul 30 14:25:35 rg-app-prod-elk systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jul 30 14:25:35 rg-app-prod-elk systemd[1]: elasticsearch.service: Unit entered failed state.
Jul 30 14:25:35 rg-app-prod-elk systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
What to do ?