ELK version 7.8.1 : Kibana took longer time to start

[Hung_M_Le]

Hi

I just recently upgraded my Elastic Docker image to use the version 7.8.1 and I notice the Kibana component took abnormally longer time to start. Thus, I observe lots of print from filebeat that It cannot connect to Kibana over the port 5601.

Please let me know if there is a work-around this issue. I provide a few snapshots of the issues because I cannot upload a text file!. Thank you in advance for your help.

Best Regards

Hung

Screen Shot 2020-08-18 at 7.05.05 PM|690x382

[warkolm (Mark Walkom)]

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

[Hung_M_Le]

Hi Mark

Is there a way to upload the log (text) file? I tried the upload option, but it only accept picture files. Please let me know and I can upload the entire log file.

Here are some extract from the log:

filebeat01 | end of elasticsearch preparation------------------------------------------------
filebeat01 | wait-for-it.sh: waiting 60 seconds for kibana:5601

filebeat01 | wait-for-it.sh: timeout occurred after waiting 60 seconds for kibana:5601
filebeat01 | wait-for-it.sh: strict mode, refusing to execute subprocess
filebeat01 | start importing kibana objects to kibana:5601
filebeat01 | % Total % Received % Xferd Average Speed Time Time Time Current
filebeat01 | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to kibana:5601; Connection refused
filebeat01 | % Total % Received % Xferd Average Speed Time Time Time Current
filebeat01 | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to kibana:5601; Connection refused
filebeat01 | % Total % Received % Xferd Average Speed Time Time Time Current
filebeat01 | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to kibana:5601; Connection refused

........

filebeat01 | % Total % Received % Xferd Average Speed Time Time Time Current
filebeat01 | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to kibana:5601; Connection refused
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:51:57Z","tags":["warning","config","deprecation"],"pid":7,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.""}
filebeat01 | % Total % Received % Xferd Average Speed Time Time Time Current
filebeat01 | Dload Upload Total Spent Left Speed
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:51:58Z","tags":["info","plugins-system"],"pid":7,"message":"Setting up [94] plugins: [taskManager,licensing,observability,eventLog,encryptedSavedObjects,code,usageCollection,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaLegacy,devTools,translations,mapsLegacy,mapsLegacyLicensing,uiActions,share,kibanaUtils,statusPage,newsfeed,kibanaReact,inspector,embeddable,advancedUiActions,embeddableEnhanced,drilldowns,esUiShared,indexPatternManagement,discover,bfetch,expressions,data,home,cloud,console,consoleExtensions,apm_oss,searchprofiler,painlessLab,grokdebugger,management,upgradeAssistant,reporting,licenseManagement,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,advancedSettings,telemetryManagementSection,fileUpload,dataEnhanced,visualizations,visTypeVega,visTypeTimelion,features,security,snapshotRestore,transform,ingestPipelines,canvas,visTypeMarkdown,visTypeTable,inputControlVis,savedObjects,navigation,lens,graph,maps,visualize,dashboard,dashboardEnhanced,savedObjectsManagement,spaces,actions,case,alerting,alertingBuiltins,ml,charts,visTypeTagcloud,visTypeVislib,visTypeTimeseries,rollup,visTypeMetric,watcher,triggers_actions_ui,siem,infra,monitoring,logstash,apm,uptime]"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:51:58Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":7,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:51:59Z","tags":["warning","plugins","security","config"],"pid":7,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:51:59Z","tags":["warning","plugins","security","config"],"pid":7,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}

kibana01 | {"type":"log","@timestamp":"2020-08-19T01:51:59Z","tags":["info","plugins","monitoring","monitoring"],"pid":7,"message":"config sourced from: production cluster"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:51:59Z","tags":["warning","plugins","monitoring","monitoring"],"pid":7,"message":"X-Pack Monitoring Cluster Alerts will not be available: undefined"}

....

kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:20Z","tags":["status","plugin:elasticsearch@7.8.1","info"],"pid":7,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:20Z","tags":["status","plugin:elasticsearch@7.8.1","info"],"pid":7,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:20Z","tags":["status","plugin:xpack_main@7.8.1","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:20Z","tags":["status","plugin:monitoring@7.8.1","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:20Z","tags":["warning","plugins","reporting"],"pid":7,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:21Z","tags":["warning","plugins","reporting"],"pid":7,"message":"Found 'server.host: "0"' in Kibana configuration. This is incompatible with Reporting. To enable Reporting to work, 'xpack.reporting.kibanaServer.hostname: 0.0.0.0' is being automatically to the configuration. You can change the setting to 'server.host: 0.0.0.0' or add 'xpack.reporting.kibanaServer.hostname: 0.0.0.0' in kibana.yml to prevent this message."}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:21Z","tags":["warning","plugins","reporting"],"pid":7,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Centos 7.8.2003 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:21Z","tags":["warning","reporting"],"pid":7,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
100 426k 100 30 100 426k 32 463k --:--:-- --:--:-- --:--:-- 463k

........

kibana01 | {"type":"log","@timestamp":"2020-08-19T01:52:40Z","tags":["info","http","server","Kibana"],"pid":7,"message":"http server running at http://0:5601"}
filebeat01 | % Total % Received % Xferd Average Speed Time Time Time Current
filebeat01 | Dload Upload Total Spent Left Speed
elasticsearch01 | {"type": "server", "timestamp": "2020-08-19T01:52:41,977Z", "level": "INFO", "component": "o.e.c.m.MetadataMappingService", "cluster.name": "docker-cluster", "node.name": "es01", "message": "[.kibana_1/pGN4ywZ9RRGRm4OJuWv2hw] update_mapping [_doc]", "cluster.uuid": "kj3hJvJEQE-lcv5VdqcWoA", "node.id": "aVVUYARoQ1u5mgS4V_mH3A" }
kibana01 | {"type":"response","@timestamp":"2020-08-19T01:52:42Z","tags":,"pid":7,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"cross-site","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,vi;q=0.8"},"remoteAddress":"172.26.0.1","userAgent":"172.26.0.1"},"res":{"statusCode":302,"responseTime":146,"contentLength":9},"message":"GET / 302 146ms - 9.0B"}
kibana01 | {"type":"response","@timestamp":"2020-08-19T01:52:42Z","tags":,"pid":7,"method":"get","statusCode":302,"req":{"url":"/spaces/enter","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"cross-site","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,vi;q=0.8"},"remoteAddress":"172.26.0.1","userAgent":"172.26.0.1"},"res":{"statusCode":302,"responseTime":164,"contentLength":9},"message":"GET /spaces/enter 302 164ms - 9.0B"}

kibana01 | {"type":"response","@timestamp":"2020-08-19T01:52:42Z","tags":,"pid":7,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"cross-site","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,vi;q=0.8"},"remoteAddress":"172.26.0.1","userAgent":"172.26.0.1"},"res":{"statusCode":200,"responseTime":386,"contentLength":9},"message":"GET /app/kibana 200 386ms - 9.0B"}

kibana01 | {"type":"response","@timestamp":"2020-08-19T01:52:43Z","tags":["api"],"pid":7,"method":"get","statusCode":200,"req":{"url":"/bundles/app/kibana/bootstrap.js","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36","accept":"/","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"http://localhost:5601/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,vi;q=0.8","if-none-match":""6a3d3aefd826738371e65890134458d3b094e00f-gzip""},"remoteAddress":"172.26.0.1","userAgent":"172.26.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":250,"contentLength":9},"message":"GET /bundles/app/kibana/bootstrap.js 200 250ms - 9.0B"}

100 426k 100 34 100 426k 8 100k 0:00:04 0:00:04 --:--:-- 100k
filebeat01 | kibana objects import status: {"success":true,"successCount":30}
kibana01 | {"type":"response","@timestamp":"2020-08-19T01:52:41Z","tags":,"pid":7,"method":"post","statusCode":200,"req":{"url":"/api/saved_objects/_import?overwrite=true","method":"post","headers":{"user-agent":"curl/7.29.0","host":"kibana:5601","accept":"/","kbn-xsrf":"true","content-length":"436441","expect":"100-continue","content-type":"multipart/form-data; boundary=----------------------------e9b360599d46"},"remoteAddress":"172.26.0.4","userAgent":"172.26.0.4"},"res":{"statusCode":200,"responseTime":4193,"contentLength":9},"message":"POST /api/saved_objects/_import?overwrite=true 200 4193ms - 9.0B"}
filebeat01 | end of kibana preparation------------------------------------------------
ilebeat; Version: 7.8.1

[warkolm (Mark Walkom)]

Use gist/pastebin/etc and link to it from a post.

[Hung_M_Le]

Here is the link to the log file. Thanks

https://gist.github.com/hmle64/cac1ba0b91351feffd230166330dd918