Ubuntu 16.04 & Elasticsearch 1.7 - Startup Error

[yashvit]

I have recently added a new node(fresh install of ubuntu & elastic install via dpkg) to my existing cluster, and randomly one server would not start elasticsearch on a restart. When manually trying to start it gives the following error:

sudo systemctl status elasticsearch

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2017-06-16 12:32:28 UTC; 3s ago
     Docs: http://www.elastic.co
  Process: 1437 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=$PID_DIR/elasticsearch.pid -Des.default.path.home=$ES_HOME -Des.default.p
 Main PID: 1437 (code=exited, status=3)

Jun 16 12:32:28 els02.xyz.net elasticsearch[1437]: java.io.FileNotFoundException: /var/run/elasticsearch/elasticsearch.pid (No such file or directory)
Jun 16 12:32:28 els02.xyz.net elasticsearch[1437]:         at java.io.FileOutputStream.open0(Native Method)
Jun 16 12:32:28 els02.xyz.net elasticsearch[1437]:         at java.io.FileOutputStream.open(FileOutputStream.java:270)
Jun 16 12:32:28 els02.xyz.net elasticsearch[1437]:         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
Jun 16 12:32:28 els02.xyz.net elasticsearch[1437]:         at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
Jun 16 12:32:28 els02.xyz.net elasticsearch[1437]:         at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:194)
Jun 16 12:32:28 els02.xyz.net elasticsearch[1437]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Jun 16 12:32:28 els02.xyz.net systemd[1]: elasticsearch.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 16 12:32:28 els02.xyz.net systemd[1]: elasticsearch.service: Unit entered failed state.
Jun 16 12:32:28 els02.xyz.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

So I went ahead and created the folder that elasticsearch requires in - /var/run/elasticsearch and changed ownership to elasticsearch:elasticsearch. After this elasticsearch starts fine. But on restart the same issue again.

Tried changing the PID_DIR folder to /var/run in the service file at /usr/lib/systemd/system/elasticsearch.service but that gives a different error:

java.io.FileNotFoundException: /var/run/elasticsearch.pid (Permission denied)

One other thing i noticed, when I manually create the dir in /var/run and start elasticsearch, it seems to work fine but the GET /_nodes does not return any stats of the OS or Filesystem. This might be a completely unrelated issue but I noticed this because I use Opserver to monitor my cluster which shows no OS/Hardware stats for this node (It was working all fine before the restart fiasco).

Linux Version: Ubuntu 16.04
Elasticsearch Version: 1.7.3

Aware of this issue which was affecting Elasticsearch 1.6 - https://github.com/elastic/elasticsearch/issues/11594