Elasticsearch failed to start yesterday

[Saplog]

Yes I will, thanks for your help

[yashvit]

Linux Version: Ubuntu 16.04
Elasticsearch Version: 1.7.3

I have recently added some new nodes to my existing cluster, and randomly one server would not start elasticsearch on restart. When manually trying to start it gives the following error:

>sudo systemctl status elasticsearch


* elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2017-06-16 12:32:28 UTC; 3s ago
     Docs: http://www.elastic.co
  Process: 1437 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=$PID_DIR/elasticsearch.pid -Des.default.path.home=$ES_HOME -Des.default.p
 Main PID: 1437 (code=exited, status=3)

Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]: java.io.FileNotFoundException: /var/run/elasticsearch/elasticsearch.pid (No such file or directory)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at java.io.FileOutputStream.open0(Native Method)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at java.io.FileOutputStream.open(FileOutputStream.java:270)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:194)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Jun 16 12:32:28 els02.jobsoid.net systemd[1]: elasticsearch.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 16 12:32:28 els02.jobsoid.net systemd[1]: elasticsearch.service: Unit entered failed state.
Jun 16 12:32:28 els02.jobsoid.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

If I create a folder /var/run/elasticsearch and change ownership to elasticsearch:elasticsearch then elasticsearch starts. On restart again the same issue.

I tried changing the PID_DIR folder to /var/run in the service file at /usr/lib/systemd/system/elasticsearch.service
with that it gives an error: java.io.FileNotFoundException: /var/run/elasticsearch.pid (Permission denied) instead of "no such file or directory"

One other thing i noticed, when I manually create the dir in /var/run and force elasticsearch to start, it seems to work fine but the GET /_nodes does not return any stats of the OS or Filesystem.

Makes me think its some permission issue?