Kibana 7.5.1 Unable to use Kibana Keystore due to Permissions Problem

[nmm (Nada)]

I installed Kibana on an Ubuntu 16.04.6 VM using a user with sudo permissions (basically, a user with root-like permissions). I have enabled and started kibana with systemctl. I have security enabled, the built-in users configured, and Kibana is connected to my 7.5.1 elasticsearch cluster which also has security enabled and the TLS transport configured. I can log into kibana with a custom user via a browser. However, I want to move the built-in kibana username and password to the kibana keystore as well as various encryption keys. To add the keys to the kibana keystore I use the same user with sudo permissions I used to install and configure Kibana. Therefore, the permissions of the keystore I created are:

opslogs@zcolo-ops-es-kibana-1:~$ ls -l /var/lib/kibana/
total 16
-rw-r--r-- 1 root root 130 Jan 14 14:45 kibana.keystore

When I add a key with "sudo" and the flag --allow-root, kibana fails to start with this error message:
{"type":"log","@timestamp":"2020-01-14T01:20:18Z","tags":["fatal","root"],"pid":1262,"message":"{ Error: "K_USER" and "K_SEC_ENCRYPTKEY" settings were not applied.
Check for spelling errors and ensure that expected plugins are installed.\n
at KbnServer._default (/usr/share/kibana/src/legacy/server/config/complete.js:90:17) code: 'InvalidConfig', processExitCode: 64 }"}

I verified the spelling by cutting and pasting K_SEC_ENCRYPTKEY. I don't even need to use the K_SEC_ENCRYPTKEY in kibana.yml file for it to fail, though it fails in both cases.

Even though the persmission are "root root" I tried to add a key to the kibana keystore without using sudo. I got this error:

opslogs@zcolo-ops-es-kibana-1:~$ /usr/share/kibana/bin/kibana-keystore list

Babel could not write cache to file: /usr/share/kibana/optimize/.babel_register_cache.json
due to a permission issue. Cache is disabled.

I experimented by changing the permission of the keystore to "kibana kibana" or "root kibana" (/var/lib/kibana/kibana.keystore), the problems as described above remained.

It seems that I need to change permissions of various files that the keystore uses so the kibana user can access them; however, I don't know what these are. The Kibana keystore documentation has no information about this. In addition, I don't have this problem with elasticsearch 7.5.1. I used an identical user with the same root-like permissions to install ES and add add TLS transport keys to its keystore with no problems. Note that ES automatically creates the keystore for me upon installation and places it in /etc/elasticsearch with permissions:
-rw-rw---- 1 root elasticsearch 395 Jan 9 17:37 elasticsearch.keystore

To my mind, manually changing permissions on a file like the kibana keystore is probably an unorthodox and problematic approach to resolving this problem, so I'd like help figuring out how to use the keystore in 7.5.1. There was an earlier question about this a few months ago, but the answer basically said I had to use the --allow-root flag and use "service kibana start" to run Kibana, or systemctl, which I have done. So I'd like some more input.

Thanks!

[jbudz (Jon Budzenski)]

Hey - we do so occasional problems with permissions like this. In the installation scripts there's a number of chown commands that we run to set permissions to the user 'kibana'.

Are you able to run as the user kibana as an alternative? Installing kibana as root should be the only needed root step. If not we can go through the process of setting kibana to run as a different user. The starting point there is to modify the unit file to run as the created user and chown all the files to the new user.

[nmm (Nada)]

Thank you for answering my question! I was beginning to think no one would.

To begin with, I am not a linux expert and I feel like I am seriously missing some linux knowledge or understanding of Linux lingo that I'm expected to know. Moreover, just to be sure, when you talk about the kibana user, I will assume you're talking about the internal, local kibana user that doesn't have a user shell and is created either at installation or by the kibana service, not the built-in kibana user that is created on an elasticsearch node with an ES script and is used for authenticating queries from the Kibana service to the elasticsearch cluster.

After installing, I enabled & started kibana with systemctl like the documentation instructed so that it's automatically started upon initialization.

This method is what I need. I didn't find any instructions on how to designate to systemd to start the Kibana service as the local user kibana upon initialization of the system without any user intervention. Changing permissions on my own seemed a bit risky, so I left it alone.

When I look at the permissions on the kibana.service file in /etc/systemd/system, it is root root with rw-r--r--.

So it looks like I am not running the kibana service as the local kibana user. The kibana service does start; however.

I'm not sure what you mean by being able to "run as the user kibana as an alternative". Meaning I need to stop kibana, disable it with systemctl, then launch a kibana instance manually? As the local kibana user? Not sure how to do that when the local kibana user doesn't have an user shell? This is one of the areas I'm confused in... I'm supposed to run kibana as the local user kibana, but the local user kibana doesn't have an user shell and I don't know what the local kibana password is, if there is one, unless it's also the build-in kibana user password? If so, then how does this work for systems that don't have security enabled but want to use the keystore?

from the password file:
kibana:x:###:###::/home/kibana:/bin/false

Thank you for helping me!

[knutster (Conrad Christensen)]

There's definitely a permission problem here, it is not possible to run kibana-keystore when installed via deb. The kibana user cannot be su or sudo su 'ed to to execute kibana-keystore and when run as root the permissions end up wrong.

An ugly workaround would be to use chroot like in the /etc/init.d/kibana startup script:

# HOME=/ chroot --userspec kibana:kibana /

You should now be able to use kibana-keystore binary.

[nmm (Nada)]

Thanks for your help! I'll check out the work around.

[nmm (Nada)]

Hi Jon Budzenski,

Would you put the chown commands you use to address my permission problem in this discussion or point me to where it's at in documentation / blog / other discussion? I'd like to compare your permissions changes to Conrad Christensen's suggestion.

Thank you!