Kibana restarts every 14seconds

b4dr4b11t · October 19, 2019, 5:10pm

Hi,

This is a newly installed Elasticsearch and Kibana on CentOS7 Core. This system was hardened following "CIS_CentOS_Linux_7_Benchmark_v2.2.0.pdf". I'm experiencing a very weird situation wherein Kibana restarts every 14 seconds, please refer to the following stackoverflow post for the session excerpts:

I posted in stackoverflow because they allow up to 30000 characters in a post while here is only 7000.
I actually checked the status of kibana service every second and it does restart at the 14th second as below:

[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 2s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 3s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 5s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 6s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 7s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 8s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 9s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 10s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 12s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 13s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 14s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           ‣ 25870 [node]

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.

I have the kibana.log but the Upload function doesn't allow me to upload .log file
Let me know how can I share the kibana.log and appreciate your help.

Thank you

b4dr4b11t · October 21, 2019, 2:35am

kibana.log can be seen/downloaded from here

tylersmalley · October 22, 2019, 10:33pm

Based on the logs, I would guess something is killing the process.

b4dr4b11t · October 30, 2019, 8:15am

Hi,

I re-installed Elasticsearch and Kibana but still getting that restart every 14 seconds. I was able to resolve this issue by removing "noexec" from /var; got the solution from Kibana server is not ready yet - throw er; // Unhandled 'error' event

I actually debugged the startup of Kibana and this was the log I got during the stop to start:

Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["status","plugin:reporting@7.4.1","info"],"pid":14119,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports: [TypeError: input.on is not a function]"}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"See Chromium's log output at \"/var/lib/kibana/headless_shell-linux/chrome_debug.log\""}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"Reporting plugin self-check failed. Please check the Kibana Reporting settings. Error: Could not close browser client handle!"}
Oct 30 15:58:17 syslog kibana: events.js:174
Oct 30 15:58:17 syslog kibana: throw er; // Unhandled 'error' event
Oct 30 15:58:17 syslog kibana: ^
Oct 30 15:58:17 syslog kibana: Error: spawn /var/lib/kibana/headless_shell-linux/headless_shell EACCES
Oct 30 15:58:17 syslog kibana: at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
Oct 30 15:58:17 syslog kibana: at onErrorNT (internal/child_process.js:415:16)
Oct 30 15:58:17 syslog kibana: at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 30 15:58:17 syslog kibana: Emitted 'error' event at:
Oct 30 15:58:17 syslog kibana: at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
Oct 30 15:58:17 syslog kibana: at onErrorNT (internal/child_process.js:415:16)
Oct 30 15:58:17 syslog kibana: at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 30 15:58:17 syslog systemd: kibana.service: main process exited, code=exited, status=1/FAILURE
Oct 30 15:58:17 syslog systemd: Unit kibana.service entered failed state.
Oct 30 15:58:17 syslog systemd: kibana.service failed.
Oct 30 15:58:17 syslog systemd: kibana.service holdoff time over, scheduling restart.
Oct 30 15:58:17 syslog systemd: Stopped Kibana.
Oct 30 15:58:17 syslog systemd: Started Kibana.
Oct 30 15:58:20 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:20Z","tags":["info","plugins-system"],"pid":14144,"message":"Setting up [4] plugins: [security,translations,inspector,data]"}
Oct 30 15:58:20 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:20Z","tags":["info","plugins","security"],"pid":14144,"message":"Setting up plugin"}

which led me to search for "kibana: Error: spawn /var/lib/kibana/headless_shell-linux/headless_shell EACCES" that directed me to the solution URL.
Thanks to Kaldayr for the solution.

Topic		Replies	Views
Kibana7.4 does not start Kibana	6	814	November 6, 2019
Kibana 6.2.3 Restarts Constantly Kibana	23	4124	May 23, 2018
Kibana 4.5.1 stop Kibana	3	1616	July 6, 2017
Why does Kibana stops after 10seconds? Kibana	13	2615	September 22, 2022
Kibana already restart Kibana	13	334	July 6, 2021

Kibana restarts every 14seconds

Related topics