Kibana server become down for 3 hrs

Elastic Stack Kibana
1

Hi All,

I have been using Kibana 7.2.0 in ES 7.2.0. The kibana became down for about 3 hrs. I have pasted kibana logs.

<--- Last few GCs --->

[4911:0x40bac70] 24520891172 ms: Mark-sweep 804.2 (1160.7) -> 804.2 (1071.7) MB, 390.7 / 0.0 ms  (average mu = 0.644, current mu = 0.000) last resort GC in old space requested
[4911:0x40bac70] 24520891565 ms: Mark-sweep 804.2 (1071.7) -> 804.2 (1018.7) MB, 393.5 / 0.0 ms  (average mu = 0.469, current mu = 0.000) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x28637dc078a1]
Security context: 0x3f87dce1e6e1 <JSObject>
    1: byteLength [0x2dab353d8219] [buffer.js:~509] [pc=0x28637e05fbb7](this=0x189b0d75f8c1 <JSFunction Buffer (sfi = 0x33bbaa913669)>,string=0x210035149f61 <Very long string[303478774]>,encoding=0x3f87dce3cc41 <String[4]: utf8>)
    2: size [0x1bcecb421831] [/usr/share/kibana/node_modules/hapi/lib/response.js:682] [bytecode=0x2b66d77e4de1 offset=71](this=0x...

{"type":"log","@timestamp":"2020-05-28T21:28:33Z","tags":["status","plugin:apm_oss@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:kibana@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:elasticsearch@7.2.0","info"],"pid":103378,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:console@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:data@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:interpreter@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:metrics@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:tile_map@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:timelion@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:ui_metric@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["status","plugin:elasticsearch@7.2.0","info"],"pid":103378,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-05-28T21:28:34Z","tags":["listening","info"],"pid":103378,"message":"Server running at http://0.0.0.0:5601"}

Can you please tell the root cause of it? Please let me know if you need more information.

Thanks,
Kata

2

Hello,

It looks like Kibana was running but couldn't access Elasticsearch. You could check the Elasticsearch logs on the machine that hosts the cluster for any issues in that time period. If there are no clues there, it could have been a network issue.

Regards,
Aaron

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.