Kibana 6.8 crashes

[jasmineL]

Hi all,

We are currently using Kibana 6.8 with basics license. And we are seeing Kibana crashes periodically. I am looking at the kiban.stdout log file and I am guessing is that related to the health check? Here are part of the logs right before the crash.

{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.120.141.44"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":302,"responseTime":3,"contentLength":9},"message":"HEAD / 302 3ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-proxy"],"pid":22320,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-service"],"pid":22320,"message":"Request will be handled by proxy GET:/."}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Keep-Alive","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":302,"responseTime":3,"contentLength":9},"message":"GET / 302 3ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-proxy"],"pid":22320,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-service"],"pid":22320,"message":"Request will be handled by proxy GET:/login.html."}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["license","debug","xpack"],"pid":22320,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":404,"req":{"url":"/properties/description.dhtml","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Close","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":404,"responseTime":54,"contentLength":9},"message":"GET /properties/description.dhtml 404 54ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-proxy"],"pid":22320,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-service"],"pid":22320,"message":"Request will be handled by proxy GET:/properties/configuration.php?tab=Status."}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["license","debug","xpack"],"pid":22320,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":404,"req":{"url":"/start.js","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Close","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":404,"responseTime":81,"contentLength":9},"message":"GET /start.js 404 81ms - 9.0B"}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":404,"req":{"url":"/login.html","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Close","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":404,"responseTime":45,"contentLength":9},"message":"GET /login.html 404 45ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-proxy"],"pid":22320,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-service"],"pid":22320,"message":"Request will be handled by proxy GET:/tmui/."}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":301,"req":{"url":"/tmui/","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Close","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":301,"responseTime":2,"contentLength":9},"message":"GET /tmui/ 301 2ms - 9.0B"}

When it is in good state,

{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["license","debug","xpack"],"pid":22320,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}

was calling once a while. But in the bad state, there are some 404 errors right after the call and then lots of this call being processed. And in just one minutes, tons of calls and logs made kibana crash....

Could anybody help? What should we do?
When we restart, Kibana is back. But this crash issue keeps happening like every 3 days....

Thanks a lot!!

[rashmi (kulkarni)]

Can you check the timestamp of that kibana.bundle.js? I was told by a developer that the file is only written to when a plugin is installed or when some other kibana file is manually changed. Anything like that happening? Also please check ES logs for more information ? More info on the package, installation - fresh or upgrade info, would help.

Thanks
Rashmi

[jasmineL]

Thanks Rashmi.
The Kibana package was build manually from git repository.
The command I am using is
node script/build.js --release --no-oss
In order to build only one package for auto deploy, we moved some cusomized plugins to legacy/core-plugins where kibana and timelion was located.
Will this be a problem?
During those every 3 days, we just normally use Kibana and no manual changes happened. And we didn't install any packages during those days. And kibana.bundle.js is fine with the old timestamp when we use to deploy.
Any more information needed?
Thanks a lot!

[jasmineL]

@rashmi, we are still seeing crashes every week. Is there any update on this topic? We changed our Kibana package from git build to the one download from Elastic website. Version is 6.8.1.

Just wonder is there any minimum environment requirement? Like memery? Space?

Thanks a lot!

[rashmi (kulkarni)]

@jbudz - any help here ?

Thanks
Rashmi

[jbudz (Jon Budzenski)]

@jasmineL

"2019-09-10T06:19:23Z","tags":["license","debug","xpack"],"pid":22320,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}

was calling once a while. But in the bad state, there are some 404 errors right after the ca

@jasmineL we'd expect an installation with those plugins to be fine with 1 cpu and 2gb of memory. Disk space for Kibana is minimal beyond the initial extraction, 1 gb should be fine. When you say crash, does the service stop entirely? Or does the status page come up? The requested routes don't make a lot of sense to me - the only html page I would expect to see is index.html. Is this Kibana installation on the internet?

[jasmineL]

Hi Jon,

When it crash, Kibana doesn't response at all. Service stop entirely. And no status page show up. From browser, it just keep connecting and later will get timeout.
We are testing the upgrade and since this crash happens every 1 week, it really block us from upgrading to 6.8....
For above errors, we even try to build aN open source version and it also crash.
Regarding the environment requirement, we are much more then the environment which should not be the problem...
We install the kibana package from https://www.elastic.co/downloads/past-releases#kibana.

Thanks.

[Christian_Dahlqvist]

You mentioned customised plugins earlier in the thread. Could you provide some more information about these?

[jasmineL]

Hi Christian,

We did have some customised plugins at the beginning. We thought about that too. After the crash issue, we reverted back and now we are trying with the pure Kibana downloaded from the website https://www.elastic.co/downloads/past-releases#kibana without any customized things. And the issue are same.. It crash after around 1 week and browser just keep connecting.

[jasmineL]

@jbudz I just talked to Chris in today's Elastic{ON} conference regarding this issue. He said you may be the expert on this area. And he is guessing it might be related to a health check memory leak issue which has been fixed in 7.x. Do you by chance know more detail on this? Could you please give us some more clue like the link to that PR?
We have 5gb memory on our kibana machine. And another team have 2gb. Since last friday, it have been crash 3 times already. Based on this info, smaller memory crash more frequently.
And as I mentioned before, by the time it crash, we are able to ssh to the kibana server and the browser is just showing 'keep connecting' without any status page or red bar showing kibana is not responding like what 5.6 do. After a while, it will just show no connection which is from the browser itself.

Thanks a lot!
Jasmine