Kibana weird fail after upgrade to 7.14.0

Elastic Stack Kibana
21

@kleckns from those logs I was not able to spot any thing that really pointed me to a possible error. Could you please navigate into http://0.0.0.0:5601 but with your browser cache disabled and confirm if you still got the same error?

22

Thank you @tiagocosta. I will try that.

I have another ES cluster in the same environment. Its the monitoring cluster to the cluster above with the Kibana issue. I spent the day stepping upgrades 7.9.3 -> 7.10.0 -> 7.11.0 -> 7.12.0 -> 7.13.0 -> 7.14.0 and making migration changes along the way. That cluster's kibana worked during each upgrade until i went from 7.13.0 to 7.14.0. so i've narrowed focus slightly.

23

part 1 of the kibana.log after i upgraded from 7.13.0 to 7.14.0 on that second cluster

{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["info","plugins-service"],"pid":22839,"message":"Plugin \"metricsEntities\" is disabled."}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, you can use the \"console\" appender in your logging configuration or define a custom one. For more details, see https://github.com/elastic/kibana/blob/master/src/core/server/logging/README.mdx"}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"\"xpack.monitoring\" is deprecated and has been replaced by \"monitoring\""}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications towork in 8.0.\""}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"\"xpack.reporting.roles\" is deprecated. Granting reporting privilege through a \"reporting_user\" role will not be supported starting in 8.0. Please set \"xpack.reporting.roles.enabled\" to \"false\" and grant reporting privileges to users using Kibana application privileges **Management > Security > Roles**."}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["info","http","server","NotReady"],"pid":22839,"message":"http server running at https://0.0.0.0:5601"}
{"type":"log","@timestamp":"2021-08-09T22:41:40+00:00","tags":["info","plugins-system"],"pid":22839,"message":"Setting up [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-08-09T22:41:40+00:00","tags":["info","plugins","taskManager"],"pid":22839,"message":"TaskManager is identified by the Kibana UUID: 1e27e9c0-270f-494a-84ee-66763e87837a"}
{"type":"log","@timestamp":"2021-08-09T22:41:50+00:00","tags":["warning","plugins","security","config"],"pid":22839,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T22:41:50+00:00","tags":["warning","plugins","reporting","config"],"pid":22839,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T22:41:50+00:00","tags":["warning","plugins","reporting","config"],"pid":22839,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.9OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-08-09T22:41:59+00:00","tags":["info","plugins","ruleRegistry"],"pid":22839,"message":"Write is disabled, not installing assets"}
{"type":"log","@timestamp":"2021-08-09T22:42:00+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-08-09T22:42:02+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 308ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 25ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 409ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 61ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 245ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 243ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 373ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 296ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 121ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 134ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 685ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 26ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 687ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 27ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] Starting to process 25 documents."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_INDEX. took: 56ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 23ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 196ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] Processed 25 documents out of 25."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 86ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 17ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 230ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] Starting to process 591 documents."}
24

part 2

{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_INDEX. took: 1025ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 666ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 50ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 19ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 43ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 20ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 138ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 750ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 353ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 356ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] Migration completed after 4111ms"}
{"type":"log","@timestamp":"2021-08-09T22:42:09+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 3621ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:09+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] Processed 591 documents out of 591."}
{"type":"log","@timestamp":"2021-08-09T22:42:09+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 50ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:09+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 7ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:10+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 197ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 1081ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 20ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 10ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 59ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 8ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 162ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:13+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 2450ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 259ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] Migration completed after 11483ms"}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","status"],"pid":22839,"message":"Kibana is now unavailable"}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","plugins-system"],"pid":22839,"message":"Starting [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":22839,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-08-09T22:42:17+00:00","tags":["info","http","server","Kibana"],"pid":22839,"message":"http server running at https://0.0.0.0:5601"}
{"type":"log","@timestamp":"2021-08-09T22:42:18+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":22839,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-08-09T22:42:18+00:00","tags":["info","plugins","securitySolution"],"pid":22839,"message":"Dependent plugin setup complete - Starting ManifestTask"}
{"type":"log","@timestamp":"2021-08-09T22:42:19+00:00","tags":["info","status"],"pid":22839,"message":"Kibana is now available (was unavailable)"}
{"type":"log","@timestamp":"2021-08-09T22:42:20+00:00","tags":["info","plugins","reporting"],"pid":22839,"message":"Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell"}
{"type":"log","@timestamp":"2021-08-09T22:42:20+00:00","tags":["warning","plugins","reporting"],"pid":22839,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
25

@kleckns looks like the migration process completed without problems and there is a log stating

{"type":"log","@timestamp":"2021-08-09T22:42:19+00:00","tags":["info","status"],"pid":22839,"message":"Kibana is now available (was unavailable)"}

Everything is pointing out that Kibana should be working as expected at https://0.0.0.0:5601

Have you tried to load the page without the browser caches enabled as I suggested?

Cheers

26

I agree with Tiago, I can't see anything wrong in these logs.

Can you enable verbose logging with:

logging:
  root:
    level: debug

After restarting Kibana, please open the chrome developer tools and then open Kibana. Please share a screenshot of the "Network" tab or even better save it as a HAR file so we can see what exactly is preventing the Kibana UI from loading.