Kibana restarts every 14seconds

Elastic Stack Kibana
1

Hi,

This is a newly installed Elasticsearch and Kibana on CentOS7 Core. This system was hardened following "CIS_CentOS_Linux_7_Benchmark_v2.2.0.pdf". I'm experiencing a very weird situation wherein Kibana restarts every 14 seconds, please refer to the following stackoverflow post for the session excerpts:

I posted in stackoverflow because they allow up to 30000 characters in a post while here is only 7000.
I actually checked the status of kibana service every second and it does restart at the 14th second as below:

[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 2s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 3s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 5s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 6s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 7s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 8s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 9s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 10s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 12s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 13s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 14s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           ‣ 25870 [node]

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.

I have the kibana.log but the Upload function doesn't allow me to upload .log file
Let me know how can I share the kibana.log and appreciate your help.

Thank you

2

kibana.log can be seen/downloaded from here

3

Based on the logs, I would guess something is killing the process.

4

Hi,

I re-installed Elasticsearch and Kibana but still getting that restart every 14 seconds. I was able to resolve this issue by removing "noexec" from /var; got the solution from Kibana server is not ready yet - throw er; // Unhandled 'error' event

I actually debugged the startup of Kibana and this was the log I got during the stop to start:

Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["status","plugin:reporting@7.4.1","info"],"pid":14119,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports: [TypeError: input.on is not a function]"}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"See Chromium's log output at \"/var/lib/kibana/headless_shell-linux/chrome_debug.log\""}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"Reporting plugin self-check failed. Please check the Kibana Reporting settings. Error: Could not close browser client handle!"}
Oct 30 15:58:17 syslog kibana: events.js:174
Oct 30 15:58:17 syslog kibana: throw er; // Unhandled 'error' event
Oct 30 15:58:17 syslog kibana: ^
Oct 30 15:58:17 syslog kibana: Error: spawn /var/lib/kibana/headless_shell-linux/headless_shell EACCES
Oct 30 15:58:17 syslog kibana: at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
Oct 30 15:58:17 syslog kibana: at onErrorNT (internal/child_process.js:415:16)
Oct 30 15:58:17 syslog kibana: at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 30 15:58:17 syslog kibana: Emitted 'error' event at:
Oct 30 15:58:17 syslog kibana: at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
Oct 30 15:58:17 syslog kibana: at onErrorNT (internal/child_process.js:415:16)
Oct 30 15:58:17 syslog kibana: at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 30 15:58:17 syslog systemd: kibana.service: main process exited, code=exited, status=1/FAILURE
Oct 30 15:58:17 syslog systemd: Unit kibana.service entered failed state.
Oct 30 15:58:17 syslog systemd: kibana.service failed.
Oct 30 15:58:17 syslog systemd: kibana.service holdoff time over, scheduling restart.
Oct 30 15:58:17 syslog systemd: Stopped Kibana.
Oct 30 15:58:17 syslog systemd: Started Kibana.
Oct 30 15:58:20 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:20Z","tags":["info","plugins-system"],"pid":14144,"message":"Setting up [4] plugins: [security,translations,inspector,data]"}
Oct 30 15:58:20 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:20Z","tags":["info","plugins","security"],"pid":14144,"message":"Setting up plugin"}

which led me to search for "kibana: Error: spawn /var/lib/kibana/headless_shell-linux/headless_shell EACCES" that directed me to the solution URL.
Thanks to Kaldayr for the solution.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.