Kibana weird fail after upgrade to 7.14.0

Elastic Stack Kibana
21

@kleckns from those logs I was not able to spot any thing that really pointed me to a possible error. Could you please navigate into http://0.0.0.0:5601 but with your browser cache disabled and confirm if you still got the same error?

22

Thank you @tiagocosta. I will try that.

I have another ES cluster in the same environment. Its the monitoring cluster to the cluster above with the Kibana issue. I spent the day stepping upgrades 7.9.3 -> 7.10.0 -> 7.11.0 -> 7.12.0 -> 7.13.0 -> 7.14.0 and making migration changes along the way. That cluster's kibana worked during each upgrade until i went from 7.13.0 to 7.14.0. so i've narrowed focus slightly.

23

part 1 of the kibana.log after i upgraded from 7.13.0 to 7.14.0 on that second cluster

{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["info","plugins-service"],"pid":22839,"message":"Plugin \"metricsEntities\" is disabled."}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, you can use the \"console\" appender in your logging configuration or define a custom one. For more details, see https://github.com/elastic/kibana/blob/master/src/core/server/logging/README.mdx"}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"\"xpack.monitoring\" is deprecated and has been replaced by \"monitoring\""}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications towork in 8.0.\""}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["warning","config","deprecation"],"pid":22839,"message":"\"xpack.reporting.roles\" is deprecated. Granting reporting privilege through a \"reporting_user\" role will not be supported starting in 8.0. Please set \"xpack.reporting.roles.enabled\" to \"false\" and grant reporting privileges to users using Kibana application privileges **Management > Security > Roles**."}
{"type":"log","@timestamp":"2021-08-09T22:41:39+00:00","tags":["info","http","server","NotReady"],"pid":22839,"message":"http server running at https://0.0.0.0:5601"}
{"type":"log","@timestamp":"2021-08-09T22:41:40+00:00","tags":["info","plugins-system"],"pid":22839,"message":"Setting up [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-08-09T22:41:40+00:00","tags":["info","plugins","taskManager"],"pid":22839,"message":"TaskManager is identified by the Kibana UUID: 1e27e9c0-270f-494a-84ee-66763e87837a"}
{"type":"log","@timestamp":"2021-08-09T22:41:50+00:00","tags":["warning","plugins","security","config"],"pid":22839,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T22:41:50+00:00","tags":["warning","plugins","reporting","config"],"pid":22839,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T22:41:50+00:00","tags":["warning","plugins","reporting","config"],"pid":22839,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.9OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-08-09T22:41:59+00:00","tags":["info","plugins","ruleRegistry"],"pid":22839,"message":"Write is disabled, not installing assets"}
{"type":"log","@timestamp":"2021-08-09T22:42:00+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-08-09T22:42:02+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 308ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 25ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 409ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> CHECK_UNKNOWN_DOCUMENTS. took: 61ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 245ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] CHECK_UNKNOWN_DOCUMENTS -> SET_SOURCE_WRITE_BLOCK. took: 243ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 373ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CALCULATE_EXCLUDE_FILTERS. took: 296ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 121ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:03+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] CALCULATE_EXCLUDE_FILTERS -> CREATE_REINDEX_TEMP. took: 134ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 685ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 26ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP_OPEN_PIT. took: 687ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_OPEN_PIT -> REINDEX_SOURCE_TO_TEMP_READ. took: 27ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] Starting to process 25 documents."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_INDEX. took: 56ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 23ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 196ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] Processed 25 documents out of 25."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 86ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:04+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 17ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 230ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] Starting to process 591 documents."}
24

part 2

{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_INDEX. took: 1025ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 666ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 50ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 19ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 43ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:05+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 20ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 138ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 750ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 353ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 356ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:06+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana_task_manager] Migration completed after 4111ms"}
{"type":"log","@timestamp":"2021-08-09T22:42:09+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_READ. took: 3621ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:09+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] Processed 591 documents out of 591."}
{"type":"log","@timestamp":"2021-08-09T22:42:09+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 50ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:09+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 7ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:10+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 197ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> REFRESH_TARGET. took: 1081ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] REFRESH_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 20ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 10ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 59ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 8ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:11+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 162ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:13+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 2450ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 259ms."}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","savedobjects-service"],"pid":22839,"message":"[.kibana] Migration completed after 11483ms"}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","status"],"pid":22839,"message":"Kibana is now unavailable"}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","plugins-system"],"pid":22839,"message":"Starting [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-08-09T22:42:14+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":22839,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-08-09T22:42:17+00:00","tags":["info","http","server","Kibana"],"pid":22839,"message":"http server running at https://0.0.0.0:5601"}
{"type":"log","@timestamp":"2021-08-09T22:42:18+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":22839,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-08-09T22:42:18+00:00","tags":["info","plugins","securitySolution"],"pid":22839,"message":"Dependent plugin setup complete - Starting ManifestTask"}
{"type":"log","@timestamp":"2021-08-09T22:42:19+00:00","tags":["info","status"],"pid":22839,"message":"Kibana is now available (was unavailable)"}
{"type":"log","@timestamp":"2021-08-09T22:42:20+00:00","tags":["info","plugins","reporting"],"pid":22839,"message":"Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell"}
{"type":"log","@timestamp":"2021-08-09T22:42:20+00:00","tags":["warning","plugins","reporting"],"pid":22839,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
25

@kleckns looks like the migration process completed without problems and there is a log stating

{"type":"log","@timestamp":"2021-08-09T22:42:19+00:00","tags":["info","status"],"pid":22839,"message":"Kibana is now available (was unavailable)"}

Everything is pointing out that Kibana should be working as expected at https://0.0.0.0:5601

Have you tried to load the page without the browser caches enabled as I suggested?

Cheers

26

I agree with Tiago, I can't see anything wrong in these logs.

Can you enable verbose logging with:

logging:
  root:
    level: debug

After restarting Kibana, please open the chrome developer tools and then open Kibana. Please share a screenshot of the "Network" tab or even better save it as a HAR file so we can see what exactly is preventing the Kibana UI from loading.

27

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.