Issue starting Kibana 7.3

Elastic Stack Kibana
1

Hi,

I have an issue bringing up Kibana 7.3
It's similar to the issues others have faced but those comments were not able to resolve the issue and I am not able to comment on that issue as it is closed due to inactivity. Hence a new post.

The issue I am seeing is pretty much the same as

But I do not know if it has been resolved.

Kibana is not able to start and crashes repeatedly.
This is a fresh install of Kibana 7.3 (one of the comments also mentions the same on a fresh install).

-- Logs begin at Fri 2019-08-09 02:00:50 CDT. --
Aug 14 17:43:28 sana-small-1.novalocal systemd[1]: Failed to start Kibana.
Aug 14 17:43:28 sana-small-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Aug 14 17:43:28 sana-small-1.novalocal systemd[1]: kibana.service failed.
Aug 15 09:53:05 sana-small-1.novalocal systemd[1]: Started Kibana.
Aug 15 09:53:06 sana-small-1.novalocal systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Aug 15 09:53:06 sana-small-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Aug 15 09:53:06 sana-small-1.novalocal systemd[1]: kibana.service failed.
Aug 15 09:53:06 sana-small-1.novalocal systemd[1]: kibana.service holdoff time over, scheduling restart.
Aug 15 09:53:06 sana-small-1.novalocal systemd[1]: Stopped Kibana.
Aug 15 09:53:06 sana-small-1.novalocal systemd[1]: Started Kibana.
Aug 15 09:53:07 sana-small-1.novalocal kibana[8303]: events.js:174
Aug 15 09:53:07 sana-small-1.novalocal kibana[8303]: throw er; // Unhandled 'error' event
Aug 15 09:53:07 sana-small-1.novalocal kibana[8303]: ^
Aug 15 09:53:07 sana-small-1.novalocal kibana[8303]: Error: EACCES: permission denied, open '/var/log/kibana'
Aug 15 09:53:07 sana-small-1.novalocal systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Aug 15 09:53:07 sana-small-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Aug 15 09:53:07 sana-small-1.novalocal systemd[1]: kibana.service failed.
Aug 15 09:53:07 sana-small-1.novalocal systemd[1]: kibana.service holdoff time over, scheduling restart.
Aug 15 09:53:07 sana-small-1.novalocal systemd[1]: Stopped Kibana.
Aug 15 09:53:07 sana-small-1.novalocal systemd[1]: Started Kibana.
Aug 15 09:53:08 sana-small-1.novalocal kibana[8317]: events.js:174
Aug 15 09:53:08 sana-small-1.novalocal systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Aug 15 09:53:08 sana-small-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Aug 15 09:53:08 sana-small-1.novalocal systemd[1]: kibana.service failed.
Aug 15 09:53:08 sana-small-1.novalocal systemd[1]: kibana.service holdoff time over, scheduling restart.
Aug 15 09:53:08 sana-small-1.novalocal systemd[1]: Stopped Kibana.
Aug 15 09:53:08 sana-small-1.novalocal systemd[1]: Started Kibana.
Aug 15 09:53:09 sana-small-1.novalocal kibana[8329]: events.js:174
Aug 15 09:53:09 sana-small-1.novalocal systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Aug 15 09:53:09 sana-small-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Aug 15 09:53:09 sana-small-1.novalocal systemd[1]: kibana.service failed.
Aug 15 09:53:09 sana-small-1.novalocal systemd[1]: kibana.service holdoff time over, scheduling restart.
Aug 15 09:53:09 sana-small-1.novalocal systemd[1]: Stopped Kibana.
Aug 15 09:53:09 sana-small-1.novalocal systemd[1]: Started Kibana.
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: events.js:174
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: throw er; // Unhandled 'error' event
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: ^
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: Error: EACCES: permission denied, open '/var/log/kibana'
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: Emitted 'error' event at:
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: at WriteStream.onerror (_stream_readable.js:691:12)
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: at WriteStream.emit (events.js:189:13)
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: at lazyFs.open (internal/fs/streams.js:272:12)
Aug 15 09:53:10 sana-small-1.novalocal kibana[8342]: at FSReqWrap.oncomplete (fs.js:141:20)
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: kibana.service failed.
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: kibana.service holdoff time over, scheduling restart.
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: Stopped Kibana.
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: start request repeated too quickly for kibana.service
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: Failed to start Kibana.
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Aug 15 09:53:10 sana-small-1.novalocal systemd[1]: kibana.service failed.

[root@sana-small-1 ~]# ls -l /var/lib/kibana/headless_shell-linux/headless_shell
-rwxr-xr-x. 1 kibana kibana 127410016 Aug 12 16:49 /var/lib/kibana/headless_shell-linux/headless_shell

So the permissions are set correctly. I wonder if this issue is due to the fact that I am running Kibana as root?

Any help to resolve this issue would be appreciated.

Thank you

2

DId you change the user and group in the kibana.service file used by systemd?

3

I did. Still the same issue.

[root@sana-small-1 ~]# ls -l /etc/systemd/system/kibana.service
-rw-r--r--. 1 kibana kibana 508 Jul 24 14:07 /etc/systemd/system/kibana.service

4

Share the contents of that file

5 
[root@sana-small-1 system]# cat kibana.service
[Unit]
Description=Kibana
StartLimitIntervalSec=30
StartLimitBurst=3

[Service]
Type=simple
User=kibana
Group=kibana
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
WorkingDirectory=/

[Install]
WantedBy=multi-user.target
[root@sana-small-1 system]#
6

The user and group here tell systemd to run kibana as kibana, not as root. Since you started kibana from the command line as root, the ownership of /var/log/kibana is probably set to root, hence running as kibana it doesn't have access.

Change these 2 values to root and you will have to add an option to the ExecStart to allow Kibana to run as root, but I don't remember the exact systax.

7

Hi,

Thank you for spotting this. I missed it.
Yes I added --allow-root to the ExecStart command. Kibana is up and running. This solves the issue.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.