Kibana 6.8 crashes

Hi all,

We are currently using Kibana 6.8 with basics license. And we are seeing Kibana crashes periodically. I am looking at the kiban.stdout log file and I am guessing is that related to the health check? Here are part of the logs right before the crash.

{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.120.141.44"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":302,"responseTime":3,"contentLength":9},"message":"HEAD / 302 3ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-proxy"],"pid":22320,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-service"],"pid":22320,"message":"Request will be handled by proxy GET:/."}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Keep-Alive","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":302,"responseTime":3,"contentLength":9},"message":"GET / 302 3ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-proxy"],"pid":22320,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-service"],"pid":22320,"message":"Request will be handled by proxy GET:/login.html."}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["license","debug","xpack"],"pid":22320,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":404,"req":{"url":"/properties/description.dhtml","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Close","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":404,"responseTime":54,"contentLength":9},"message":"GET /properties/description.dhtml 404 54ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-proxy"],"pid":22320,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-service"],"pid":22320,"message":"Request will be handled by proxy GET:/properties/configuration.php?tab=Status."}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["license","debug","xpack"],"pid":22320,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":404,"req":{"url":"/start.js","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Close","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":404,"responseTime":81,"contentLength":9},"message":"GET /start.js 404 81ms - 9.0B"}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":404,"req":{"url":"/login.html","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Close","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":404,"responseTime":45,"contentLength":9},"message":"GET /login.html 404 45ms - 9.0B"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-proxy"],"pid":22320,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["debug","legacy-service"],"pid":22320,"message":"Request will be handled by proxy GET:/tmui/."}
{"type":"response","@timestamp":"2019-09-10T06:19:23Z","tags":[],"pid":22320,"method":"get","statusCode":301,"req":{"url":"/tmui/","method":"get","headers":{"host":"10.120.141.44:5601","accept-charset":"iso-8859-1,utf-8;q=0.9,*;q=0.1","accept-language":"en","connection":"Close","user-agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)","pragma":"no-cache","accept":"image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*"},"remoteAddress":"10.120.229.134","userAgent":"10.120.229.134"},"res":{"statusCode":301,"responseTime":2,"contentLength":9},"message":"GET /tmui/ 301 2ms - 9.0B"}

When it is in good state,

{"type":"log","@timestamp":"2019-09-10T06:19:23Z","tags":["license","debug","xpack"],"pid":22320,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}

was calling once a while. But in the bad state, there are some 404 errors right after the call and then lots of this call being processed. And in just one minutes, tons of calls and logs made kibana crash....

Could anybody help? What should we do?
When we restart, Kibana is back. But this crash issue keeps happening like every 3 days....

Thanks a lot!!

Can you check the timestamp of that kibana.bundle.js? I was told by a developer that the file is only written to when a plugin is installed or when some other kibana file is manually changed. Anything like that happening? Also please check ES logs for more information ? More info on the package, installation - fresh or upgrade info, would help.

Thanks
Rashmi

Thanks Rashmi.
The Kibana package was build manually from git repository.
The command I am using is
node script/build.js --release --no-oss
In order to build only one package for auto deploy, we moved some cusomized plugins to legacy/core-plugins where kibana and timelion was located.
Will this be a problem?
During those every 3 days, we just normally use Kibana and no manual changes happened. And we didn't install any packages during those days. And kibana.bundle.js is fine with the old timestamp when we use to deploy.
Any more information needed?
Thanks a lot!