Kibana fails to start after increasing disk space!

Elastic Stack Kibana
1

Hello,

I am having problems with Kibana. My ELK stack was up and running smoothly, until I realized it was running out of disk space. Then after increasing the disk space, I see 502 Bad Gatewy. Now after checking the status, I can see ES and LS are running, but Kibana is failing to start with the following message:

root@ELK:~# service kibana status
● kibana.service - no description given
   Loaded: loaded (/lib/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Thu 2016-06-16 14:01:26 CEST; 3min 43s ago
  Process: 1631 ExecStart=/opt/kibana/bin/kibana (code=exited, status=1/FAILURE)
 Main PID: 1631 (code=exited, status=1/FAILURE)

Jun 16 14:01:25 ELK systemd[1]: kibana.service: Unit entered failed state.
Jun 16 14:01:25 ELK systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 16 14:01:26 ELK systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Jun 16 14:01:26 ELK systemd[1]: Stopped no description given.
Jun 16 14:01:26 ELK systemd[1]: kibana.service: Start request repeated too quickly.
Jun 16 14:01:26 ELK systemd[1]: Failed to start no description given.

Now, I also don't have Kibana directory in /var/log/. Can someone please help me?

Regards,
Iqbal

2

Hi Iqbal,

If you're system is using systemd then I think your log messages are here; journalctl -u kibana.service

If that doesn't show you what the problem is, you might have to uninstall and reinstall Kibana.

Save your kibana.yml file before uninstalling if you have changed any settings in it
(its probably at /opt/kibana/config/kibana.yml)

The good news is that if your .kibana index in your Elasticsearch is OK you won't lose any of your stuff.

OR.. you could install Kibana (of the same version) on another machine and point to your Elasticsearch just to see that everything works before you uninstall or try to fix the broken one.

Regards,
Lee

3

curl -XDELETE 'http://localhost:9200/winlogbeat-*/' has done the trick for me. I recently installed Winlogbeat and it was sending huge amount of logs to make the /root directory of ELK server full.