Kibana restarts every 14seconds

b4dr4b11t · October 19, 2019, 5:10pm

Hi,

This is a newly installed Elasticsearch and Kibana on CentOS7 Core. This system was hardened following "CIS_CentOS_Linux_7_Benchmark_v2.2.0.pdf". I'm experiencing a very weird situation wherein Kibana restarts every 14 seconds, please refer to the following stackoverflow post for the session excerpts:

I posted in stackoverflow because they allow up to 30000 characters in a post while here is only 7000.
I actually checked the status of kibana service every second and it does restart at the 14th second as below:

[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 2s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 3s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 5s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]#
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 6s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 7s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 8s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 9s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 10s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 12s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 13s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           └─25870 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.
[root@syslog ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-10-20 08:18:55 +08; 14s ago
 Main PID: 25870 (node)
   CGroup: /system.slice/kibana.service
           ‣ 25870 [node]

Oct 20 08:18:55 syslog systemd[1]: Started Kibana.

I have the kibana.log but the Upload function doesn't allow me to upload .log file
Let me know how can I share the kibana.log and appreciate your help.

Thank you

b4dr4b11t · October 21, 2019, 2:35am

kibana.log can be seen/downloaded from here

tylersmalley · October 22, 2019, 10:33pm

Based on the logs, I would guess something is killing the process.

b4dr4b11t · October 30, 2019, 8:15am

Hi,

I re-installed Elasticsearch and Kibana but still getting that restart every 14 seconds. I was able to resolve this issue by removing "noexec" from /var; got the solution from Kibana server is not ready yet - throw er; // Unhandled 'error' event

I actually debugged the startup of Kibana and this was the log I got during the stop to start:

Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["status","plugin:reporting@7.4.1","info"],"pid":14119,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"The Reporting plugin encountered issues launching Chromium in a self-test. You may have trouble generating reports: [TypeError: input.on is not a function]"}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"See Chromium's log output at \"/var/lib/kibana/headless_shell-linux/chrome_debug.log\""}
Oct 30 15:58:16 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:16Z","tags":["reporting","warning"],"pid":14119,"message":"Reporting plugin self-check failed. Please check the Kibana Reporting settings. Error: Could not close browser client handle!"}
Oct 30 15:58:17 syslog kibana: events.js:174
Oct 30 15:58:17 syslog kibana: throw er; // Unhandled 'error' event
Oct 30 15:58:17 syslog kibana: ^
Oct 30 15:58:17 syslog kibana: Error: spawn /var/lib/kibana/headless_shell-linux/headless_shell EACCES
Oct 30 15:58:17 syslog kibana: at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
Oct 30 15:58:17 syslog kibana: at onErrorNT (internal/child_process.js:415:16)
Oct 30 15:58:17 syslog kibana: at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 30 15:58:17 syslog kibana: Emitted 'error' event at:
Oct 30 15:58:17 syslog kibana: at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
Oct 30 15:58:17 syslog kibana: at onErrorNT (internal/child_process.js:415:16)
Oct 30 15:58:17 syslog kibana: at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 30 15:58:17 syslog systemd: kibana.service: main process exited, code=exited, status=1/FAILURE
Oct 30 15:58:17 syslog systemd: Unit kibana.service entered failed state.
Oct 30 15:58:17 syslog systemd: kibana.service failed.
Oct 30 15:58:17 syslog systemd: kibana.service holdoff time over, scheduling restart.
Oct 30 15:58:17 syslog systemd: Stopped Kibana.
Oct 30 15:58:17 syslog systemd: Started Kibana.
Oct 30 15:58:20 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:20Z","tags":["info","plugins-system"],"pid":14144,"message":"Setting up [4] plugins: [security,translations,inspector,data]"}
Oct 30 15:58:20 syslog kibana: {"type":"log","@timestamp":"2019-10-30T07:58:20Z","tags":["info","plugins","security"],"pid":14144,"message":"Setting up plugin"}

which led me to search for "kibana: Error: spawn /var/lib/kibana/headless_shell-linux/headless_shell EACCES" that directed me to the solution URL.
Thanks to Kaldayr for the solution.

system · November 27, 2019, 8:15am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana service restarting after few seconds Kibana	25	3513	May 21, 2021
Kibana - Request Timeout after 30000ms at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15 Kibana	3	5533	July 4, 2019
Kibana Service Not Stopping Kibana	10	2006	September 17, 2018
Kibana 6.2.3 Restarts Constantly Kibana	23	3922	May 23, 2018
Kibana Crashes Periodically Kibana	9	2030	July 17, 2019

Kibana restarts every 14seconds

Related topics