Unable to start kibana with systemd on ubuntu while manually does

[Blason (R)]

Hi Team,

I am facing a weird isssue. On my ubuntu system I am unable to start kibana process with systemd and its failing while if I am starting from command line or manually it does.

Any clud why? below are the error messages consistently appearing when I try to start using systemd unit

Jun 10 14:38:17 xxxx kibana[3015650]:  FATAL  Error: EACCES: permission denied, mkdir '/usr/share/kibana/data'
Jun 10 14:38:17 xxxx systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jun 10 14:38:17 xxxx systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 10 14:38:17 xxxx audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kibana comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jun 10 14:38:20 xxxx audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kibana comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 10 14:38:20 xxxx audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kibana comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 10 14:38:20 xxxx systemd[1]: kibana.service: Scheduled restart job, restart counter is at 1.
Jun 10 14:38:20 xxxx audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kibana comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 10 14:38:22 xxxx kibana[3015681]:  FATAL  Error: EACCES: permission denied, mkdir '/usr/share/kibana/data'
Jun 10 14:38:22 xxxx systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jun 10 14:38:22 xxxx systemd[1]: kibana.service: Failed with result 'exit-code'.

[stephenb (Stephen Brown)]

What are the permissions on

/usr/share/kibana/data
And
/usr/share/kibana

Perhaps the permissions are not correct.

When you start with systemd Kibana runs as the Kibana user, so if somewhere along the line you started manually or created that directory with another user like root , Kibana may not be able to access that directory.

[Blason (R)]

@stephenb

/usr/share/kibana

I believe /usr/share/kibana/data directory does exist? I verfied other installations as well and no such directory exists

root@xxxx:/home/dnsbsiemadmin# ls -l /usr/share/kibana
total 1444
drwxr-xr-x   2 root root    4096 Mar  9 14:23 bin
-rw-r--r--   1 root root    3860 Feb 24 04:22 LICENSE.txt
drwxr-xr-x   6 root root    4096 Mar  9 14:23 node
drwxr-xr-x 824 root root   28672 Mar  9 14:23 node_modules
-rw-r--r--   1 root root 1415584 Feb 24 04:22 NOTICE.txt
-rw-r--r--   1 root root     740 Feb 24 04:22 package.json
drwxr-xr-x   2 root root    4096 Feb 24 04:22 plugins
-rw-r--r--   1 root root    3970 Feb 24 04:22 README.txt
drwxr-xr-x   9 root root    4096 Mar  9 14:23 src
drwxr-xr-x   3 root root    4096 Mar  9 14:23 x-pack
root@xxxx:/home/dnsbsiemadmin# ls -l /usr/share/kibana/data
ls: cannot access '/usr/share/kibana/data': No such file or directory

[stephenb (Stephen Brown)]

What version and how did you install?

Kibana Data per the docs should be in

data
The location of the data files written to disk by Kibana and its plugins /var/lib/kibana path.data

sbrown@stephenb-metrics-test-es1:/usr/share$ sudo ls -la /var/lib/kibana
total 24
drwxr-s---  4 kibana kibana 4096 Oct 23  2021 .
drwxr-xr-x 41 root   root   4096 Oct 26  2021 ..
drwxr-xr-x  4 kibana kibana 4096 May 10  2020 headless_shell-linux
-rw-r--r--  1 kibana kibana  178 Aug 16  2021 kibana.keystore.bak
drwxr-xr-x  2 kibana kibana 4096 May 10  2020 optimize
-rw-r--r--  1 kibana kibana   36 May 10  2020 uuid
sbrown@stephenb-metrics-test-es1:/usr/share$ 

Note the permissions / users, why your install is looking / trying to put data in the usr/share/kibana directory I am unclear. But the error makes a bit sense ... when started with systemd Kibana runs as the kibana user so it would not have permissions to read and write in the /usr/share/kibana directory.

Perhaps you somehow changed path.data

Perhaps clean up and reinstall.