I dont know what happened all of a sudden but Kibana keeps restarting then fails after the 3rd restart. there aren't any hints in the output logs that could show me why its failing to start:
kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-10-28 18:20:55 UTC; 5s ago
Main PID: 27429 (node)
Tasks: 11 (limit: 4915)
CGroup: /system.slice/kibana.service
└─27429 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["debug","plugins","data"],"pid":27429,"message":"Initializing plugin"}
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["info","plugins","data"],"pid":27429,"message":"Setting up plugin"}
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["debug","root"],"pid":27429,"message":"starting root"}
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["debug","plugins-service"],"pid":27429,"message":"Plugins service starts
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["info","plugins-system"],"pid":27429,"message":"Starting [3] plugins: [se
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["debug","plugins-system"],"pid":27429,"message":"Starting plugin \"securi
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["debug","plugins","security"],"pid":27429,"message":"Starting plugin"}
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["debug","plugins-system"],"pid":27429,"message":"Starting plugin \"transl
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["debug","plugins-system"],"pid":27429,"message":"Starting plugin \"data\"
Oct 28 18:20:58 ip-172-31-24-70 kibana[27429]: {"type":"log","@timestamp":"2019-10-28T18:20:58Z","tags":["debug","legacy-service"],"pid":27429,"message":"starting legacy service"
root@ip-172-31-24-70:/etc/elasticsearch# service kibana status
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-10-28 18:21:05 UTC; 4ms ago
Process: 27429 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=64)
Main PID: 27429 (code=exited, status=64)
Oct 28 18:21:05 ip-172-31-24-70 systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Oct 28 18:21:05 ip-172-31-24-70 systemd[1]: kibana.service: Scheduled restart job, restart counter is at 9.
Oct 28 18:21:05 ip-172-31-24-70 systemd[1]: Stopped Kibana.
Oct 28 18:21:05 ip-172-31-24-70 systemd[1]: kibana.service: Start request repeated too quickly.
Oct 28 18:21:05 ip-172-31-24-70 systemd[1]: kibana.service: Failed with result 'exit-code'.
Oct 28 18:21:05 ip-172-31-24-70 systemd[1]: Failed to start Kibana.
Additional Logs:
Oct 28 18:28:25 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:25Z","tags":["debug","plugins-discovery"],"pid":27691,"message":"Successfully discovered plugin \"security\" at \"/usr/share/kibana/x-pack/plugins/security\""}
Oct 28 18:28:25 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:25Z","tags":["debug","plugins-discovery"],"pid":27691,"message":"Successfully discovered plugin \"translations\" at \"/usr/share/kibana/x-pack/plugins/translations\""}
Oct 28 18:28:25 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:25Z","tags":["debug","plugins","data"],"pid":27691,"message":"\"/usr/share/kibana/src/plugins/data/server\" does not export \"config\"."}
Oct 28 18:28:25 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:25Z","tags":["debug","config"],"pid":27691,"message":"Marking config path as handled: xpack,security"}
Oct 28 18:28:26 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:26Z","tags":["debug","plugins","translations"],"pid":27691,"message":"\"/usr/share/kibana/x-pack/plugins/translations/server\" does not export \"config\"."}
Oct 28 18:28:26 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:26Z","tags":["debug","plugins-service"],"pid":27691,"message":"Discovered 4 plugins."}
Oct 28 18:28:26 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:26Z","tags":["debug","http"],"pid":27691,"message":"starting NotReady server"}
Oct 28 18:28:26 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:26Z","tags":["debug","elasticsearch-service"],"pid":27691,"message":"Setting up elasticsearch service"}
Oct 28 18:28:26 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:26Z","tags":["debug","elasticsearch-service"],"pid":27691,"message":"Creating elasticsearch clients"}
Oct 28 18:28:26 ip-172-31-24-70 kibana[27691]: {"type":"log","@timestamp":"2019-10-28T18:28:26Z","tags":["debug","plugins-service"],"pid":27691,"message":"Setting up plugins service"}
Can you guys pleeeeease help its driving me crazy.
Any help or guidance is greatly appreciated.
Thanks!