Kibana service will not start Exit 126

Elastic Stack Kibana
1

Upon trying to start kibana 7.16.3 after a fresh successful install on rhel 8, I am getting the following errors. Any ideas on what the cause could be. Thank you so much for any help.

> [root@attack-range-elastic-server vagrant]# sudo -u kibana /usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid &
> [1] 8636
> [root@attack-range-elastic-server vagrant]# /usr/share/kibana/bin/kibana: line 29: /usr/share/kibana/bin/../node/bin/node: Operation not permitted
> /usr/share/kibana/bin/kibana: line 29: /usr/share/kibana/bin/../node/bin/node: Success
> 
> [1]+  Exit 126                sudo -u kibana /usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid
> [root@attack-range-elastic-server vagrant]# systemctl start kibana
> [root@attack-range-elastic-server vagrant]# systemctl status kibana
> ● kibana.service - Kibana
>    Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
>    Active: failed (Result: exit-code) since Mon 2022-01-24 16:48:33 EST; 2s ago
>      Docs: https://www.elastic.co
>   Process: 8656 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid (code=exited, status=126)
>  Main PID: 8656 (code=exited, status=126)
> 
> Jan 24 16:48:30 attack-range-elastic-server systemd[1]: kibana.service: Main process exited, code=exited, status=126/n/a
> Jan 24 16:48:30 attack-range-elastic-server systemd[1]: kibana.service: Failed with result 'exit-code'.
> Jan 24 16:48:33 attack-range-elastic-server systemd[1]: kibana.service: Service RestartSec=3s expired, scheduling restart.
> Jan 24 16:48:33 attack-range-elastic-server systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
> Jan 24 16:48:33 attack-range-elastic-server systemd[1]: Stopped Kibana.
> Jan 24 16:48:33 attack-range-elastic-server systemd[1]: kibana.service: Start request repeated too quickly.
> Jan 24 16:48:33 attack-range-elastic-server systemd[1]: kibana.service: Failed with result 'exit-code'.
> Jan 24 16:48:33 attack-range-elastic-server systemd[1]: Failed to start Kibana.
> [root@attack-range-elastic-server vagrant]#
2

Can you check /var/log/kibana/kibana.log and post that?

3

That file is present but empty

4

Exit code 126 means there is a permission problem, or the command is not an executable file. This message seems to agree: line 29: /usr/share/kibana/bin/../node/bin/node: Operation not permitted

Does the kibana user have permission to read all the files that have been installed with Kibana, and to execute all of the binaries?

5

Thanks @tsullivan . Yes, I've verified that the kibana user has rights. In my troubleshooting, I even did a chmod 777 against the whole /usr/share/kibana directory just as a test and still get the same error.

6

Actually I think that might be my issue after all. Although, in looking at the file permissions I am not sure why the "vagrant" user can execute node but my "kibana" user cannot. The file permissions should have full rights for everyone?

[vagrant@attack-range-elastic-server ~]$ sudo -u kibana /usr/share/kibana/bin/../node/bin/node
sudo: unable to execute /usr/share/kibana/bin/../node/bin/node: Operation not permitted
[vagrant@attack-range-elastic-server ~]$ /usr/share/kibana/bin/../node/bin/node
Welcome to Node.js v16.13.0.
Type ".help" for more information.
>
(To exit, press Ctrl+C again or Ctrl+D or type .exit)
>
[vagrant@attack-range-elastic-server ~]$ ls -la /usr/share/kibana/bin/../node/bin/node
-rwxrwxrwx. 1 kibana kibana 79303064 Jan  6 18:42 /usr/share/kibana/bin/../node/bin/node
7

Looks like this actually might be related to RHEL fapolicy in case anyone else runs into this behavior.