Kibana not running as a service

Hello everyone,

I've installed ELK stack on an Ubuntu Server 22.04.

My problem is that I've start elasticsearch with "systemctl start elasticsearch.service" because bin/elasticsearch does not work, I always have an error and my kibana is running with "bin/kibana".
It seems that they're communicating well but when I'm checking kibana status with "systemctl status kibana", it shows that it failed to start but it's good running because I can access on browser.
So I have few questions :
-What's the difference between these 2 starting commands "bin/kibana" and "systemctl start kibana"
-Why does it works with "systemctl start elasticsearch.service" and "bin/kibana"
-Why does "systemctl status kibana" shows that kibana is not running ?

Hi,

I suppose you installed elasticsearch and kibana in different ways.
You may installed Elasticsearch with Debian Package and installed Kibana from tar.gz archive. If you install kibana with Debian Package, systemctl will work.

I installed both with debian packages, first time I started elasticsearch was "systemctl start" because bin/elastic doesn't work for a weird reason and first time i started kibana was bin/kibana --allow-root with a setup before.
If i stop kibana by ctrl+c in the terminal and use systemctl start kibana, it"works" but Kibana is not joinable on my browser.
I don't understand why because the setup goes well and bin/kibana still launches kibana

How did you configure kibana? Where is the kibana.yml and your current directory of the terminal?

Yes, kibana has been configured.
Yml files are in /etc/kibana and /etc/elasticsearch

Installed with deb package, default Kibana home directory is /usr/share/kibana.
Binary scripts is at /usr/share/kibana/bin/kibana. If current directory of your terminal is /usr/share/kibana, bin/kibana will start Kibana but bin/elasticsearch should not work.

systemd is a program to manage services and systemctl is a command to control systemd.
systemd manages services according to the unit files: /lib/systemd/system/kibana.service overridden by /etc/systemd/system/kibana.service.

If you have started Kibana outside of systemd (directly start by bin/kibana ), systemctl status kibana cound not catch Kibana processes outside of systemd.

I'm not sure Kibana allows multiple instances from single install. Could you try stop the kibana process started from the command line, and start only from systemctl?

If it still fails to start, please share the log from
systemctl status kibana -l
journalctl -u kibana

My binaries scripts are actually in /usr/share/kibana and /usr/share/elasticsearch but yml are in /etc/.. I did not change theses default directories.


First, please copy the logs as text not as picture for someone copyable and searcheable.

Can you please check ls -l /var/log/kibana/ for permissions?

One possibility is the log files are created by some user account at first execution from bin/kibana and they are not accessible from kibana user used by systemctl.

If then, simple solution is uninstall and reinstall kibana, start it from systemctl first.

Here is my journalctl :

root@odin:/usr/share/kibana# journalctl -u kibana mars 27 15:24:23 odin systemd[1]: Started Kibana. mars 27 15:24:27 odin kibana[2658]: [2023-03-27T13:24:26.921+00:00][INFO ][node> mars 27 15:24:27 odin kibana[2658]: [Error: EACCES: permission denied, open '/v> mars 27 15:24:27 odin kibana[2658]: errno: -13, mars 27 15:24:27 odin kibana[2658]: code: 'EACCES', mars 27 15:24:27 odin kibana[2658]: syscall: 'open', mars 27 15:24:27 odin kibana[2658]: path: '/var/log/kibana/kibana.log' mars 27 15:24:27 odin kibana[2658]: } mars 27 15:24:27 odin systemd[1]: kibana.service: Main process exited, code=exi> mars 27 15:24:27 odin systemd[1]: kibana.service: Failed with result 'exit-code> mars 27 15:24:27 odin systemd[1]: kibana.service: Consumed 2.679s CPU time. mars 27 15:24:30 odin systemd[1]: kibana.service: Scheduled restart job, restar> mars 27 15:24:30 odin systemd[1]: Stopped Kibana. mars 27 15:24:30 odin systemd[1]: kibana.service: Consumed 2.679s CPU time. mars 27 15:24:30 odin systemd[1]: Started Kibana. mars 27 15:24:32 odin kibana[2677]: [2023-03-27T13:24:31.997+00:00][INFO ][node> mars 27 15:24:32 odin kibana[2677]: [Error: EACCES: permission denied, open '/v> mars 27 15:24:32 odin kibana[2677]: errno: -13, mars 27 15:24:32 odin kibana[2677]: code: 'EACCES', mars 27 15:24:32 odin kibana[2677]: syscall: 'open', mars 27 15:24:32 odin kibana[2677]: path: '/var/log/kibana/kibana.log' mars 27 15:24:32 odin kibana[2677]: } mars 27 15:24:32 odin systemd[1]: kibana.service: Main process exited, code=exi> lines 1-23...skipping... mars 27 15:24:23 odin systemd[1]: Started Kibana. mars 27 15:24:27 odin kibana[2658]: [2023-03-27T13:24:26.921+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui] mars 27 15:24:27 odin kibana[2658]: [Error: EACCES: permission denied, open '/var/log/kibana/kibana.log'] { mars 27 15:24:27 odin kibana[2658]: errno: -13, mars 27 15:24:27 odin kibana[2658]: code: 'EACCES', mars 27 15:24:27 odin kibana[2658]: syscall: 'open', mars 27 15:24:27 odin kibana[2658]: path: '/var/log/kibana/kibana.log' mars 27 15:24:27 odin kibana[2658]: } mars 27 15:24:27 odin systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE mars 27 15:24:27 odin systemd[1]: kibana.service: Failed with result 'exit-code'. mars 27 15:24:27 odin systemd[1]: kibana.service: Consumed 2.679s CPU time. mars 27 15:24:30 odin systemd[1]: kibana.service: Scheduled restart job, restart counter is at 1. mars 27 15:24:30 odin systemd[1]: Stopped Kibana. mars 27 15:24:30 odin systemd[1]: kibana.service: Consumed 2.679s CPU time. mars 27 15:24:30 odin systemd[1]: Started Kibana. mars 27 15:24:32 odin kibana[2677]: [2023-03-27T13:24:31.997+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui] mars 27 15:24:32 odin kibana[2677]: [Error: EACCES: permission denied, open '/var/log/kibana/kibana.log'] { mars 27 15:24:32 odin kibana[2677]: errno: -13, mars 27 15:24:32 odin kibana[2677]: code: 'EACCES', mars 27 15:24:32 odin kibana[2677]: syscall: 'open', mars 27 15:24:32 odin kibana[2677]: path: '/var/log/kibana/kibana.log' mars 27 15:24:32 odin kibana[2677]: } mars 27 15:24:32 odin systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE mars 27 15:24:32 odin systemd[1]: kibana.service: Failed with result 'exit-code'. mars 27 15:24:32 odin systemd[1]: kibana.service: Consumed 2.006s CPU time. mars 27 15:24:35 odin systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2. mars 27 15:24:35 odin systemd[1]: Stopped Kibana. mars 27 15:24:35 odin systemd[1]: kibana.service: Consumed 2.006s CPU time. mars 27 15:24:35 odin systemd[1]: Started Kibana. mars 27 15:24:36 odin kibana[2695]: [2023-03-27T13:24:36.748+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui] mars 27 15:24:36 odin kibana[2695]: [Error: EACCES: permission denied, open '/var/log/kibana/kibana.log'] { mars 27 15:24:36 odin kibana[2695]: errno: -13, mars 27 15:24:36 odin kibana[2695]: code: 'EACCES', mars 27 15:24:36 odin kibana[2695]: syscall: 'open', mars 27 15:24:36 odin kibana[2695]: path: '/var/log/kibana/kibana.log' mars 27 15:24:36 odin kibana[2695]: } mars 27 15:24:36 odin systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE mars 27 15:24:36 odin systemd[1]: kibana.service: Failed with result 'exit-code'. mars 27 15:24:36 odin systemd[1]: kibana.service: Consumed 1.908s CPU time. mars 27 15:24:39 odin systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3. mars 27 15:24:39 odin systemd[1]: Stopped Kibana. mars 27 15:24:39 odin systemd[1]: kibana.service: Consumed 1.908s CPU time. mars 27 15:24:39 odin systemd[1]: kibana.service: Start request repeated too quickly. mars 27 15:24:39 odin systemd[1]: kibana.service: Failed with result 'exit-code'. mars 27 15:24:39 odin systemd[1]: Failed to start Kibana. mars 30 09:20:44 odin systemd[1]: Started Kibana. mars 30 09:20:46 odin kibana[6057]: [2023-03-30T09:20:46.972+02:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui] mars 30 09:20:47 odin kibana[6057]: [Error: EACCES: permission denied, open '/var/log/kibana/kibana.log'] { mars 30 09:20:47 odin kibana[6057]: errno: -13,

About the permissions, I always have worked as root, so if something has been created, it's by root.
Is it a problem if I'm using systemctl start elasticsearch and bin/kibana ?
I mean, it works so I was just wondering if it's clean to use 2 types of starting process

I've set up xpack security, maybe permissions is denied because my kibana.yml has no elastic user and password binded ?
bin/kibana-setup --enrollment-token .. works without elastic user binded.

Ok so, I tried to use an old snapshot and install elasticsearch + kibana and start both with systemctl
I've just setup kibana with bin/kibana-setup but it looks like it works

Hmm. The log said "Error: EACCES: permission denied, open '/var/log/kibana/kibana.log'". It means it is definitely the problem of permission of ubuntu file system. It is not the problem of user permissions in elasticsearch.

That's why the problem happened. '/var/log/kibana/kibana.log' was made by root and not accessible from kibana user which systemctl use for kibana process.

I suppose it is not an intended use just as problem happened to you.

Thank you for all of your replies and help, now it works, i'll use systemctl for both, It's more clean

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.