Upgrading Kibana to 7.10.2 hangs while migrating

Elastic Stack Kibana
1

Opening a new thread as suggested by @rudolf.

So I've restored .kibana_* from a snapshot and started a 7.10.2 kibana server that in turn starts the migration. These are the logs it producses:


Jan 26 08:03:50 ip-10-164-120-181 systemd: Started Kibana.
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["info","plugins-service"],"pid":20806,"message":"Plugin \"ml\" is disabled."}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["info","plugins-service"],"pid":20806,"message":"Plugin \"visTypeXy\" is disabled."}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["info","plugins-service"],"pid":20806,"message":"Plugin \"auditTrail\" is disabled."}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["info","plugins-service"],"pid":20806,"message":"Plugin \"graph\" is disabled."}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["info","plugins-service"],"pid":20806,"message":"Plugin \"licenseManagement\" is disabled."}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["info","plugins-service"],"pid":20806,"message":"Plugin \"security\" is disabled."}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["warning","config","deprecation"],"pid":20806,"message":"kibana.defaultAppId is deprecated and will be removed in 8.0. Please use the `defaultRoute` advanced setting instead"}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["warning","config","deprecation"],"pid":20806,"message":"\"xpack.monitoring\" is deprecated and has been replaced by \"monitoring\""}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["warning","config","deprecation"],"pid":20806,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["info","plugins-system"],"pid":20806,"message":"Setting up [92] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,encryptedSavedObjects,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,dashboardMode,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,snapshotRestore,maps,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,charts,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,securitySolution,infra,monitoring,logstash,apm,uptime,bfetch,canvas]"}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":20806,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["warning","plugins","reporting","config"],"pid":20806,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["warning","plugins","reporting","config"],"pid":20806,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.7.1908 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["warning","plugins","actions","actions"],"pid":20806,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
Jan 26 08:03:56 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:56Z","tags":["warning","plugins","alerts","plugins","alerting"],"pid":20806,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["info","plugins","monitoring","monitoring"],"pid":20806,"message":"config sourced from: production cluster"}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Starting saved objects migrations"}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Creating index .kibana_task_manager_2."}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["error","elasticsearch","data"],"pid":20806,"message":"[resource_already_exists_exception]: index [.kibana_task_manager_2/XFH-XmCfQIOCqswg3A93bA] already exists"}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["warning","savedobjects-service"],"pid":20806,"message":"Unable to connect to Elasticsearch. Error: resource_already_exists_exception"}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["warning","savedobjects-service"],"pid":20806,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_2 and restarting Kibana."}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Creating index .kibana_8."}
Jan 26 08:03:57 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:57Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Migrating .kibana_7 saved objects to .kibana_8"}
Jan 26 08:03:58 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:03:58Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:00 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:00Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:00 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:00Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Pointing alias .kibana to .kibana_8."}
Jan 26 08:04:00 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:00Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Finished in 3339ms."}
Jan 26 08:04:01 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:01Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:03 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:03Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:04 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:04Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:06 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:06Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:07 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:07Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:09 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:09Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:11 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:11Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:12 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:12Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:14 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:14Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:15 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:15Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:17 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:17Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:18 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:18Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:20 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:20Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:21 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:21Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:23 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:23Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:24 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:24Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:26 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:26Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:27 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:27Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:29 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:29Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:30 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:30Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
Jan 26 08:04:32 ip-10-164-120-181 kibana: {"type":"log","@timestamp":"2021-01-26T07:04:32Z","tags":["info","savedobjects-service"],"pid":20806,"message":"Detected mapping change in \"properties.originId\""}
2

It seems like Kibana doesn't start up because there is already a migration "lock" on the .kibana_task_manager index. So the migration for .kibana succeeds but Kibana still can't start up. If you're restoring from a known working snapshot and followed these instructions this should not happen https://www.elastic.co/guide/en/kibana/7.10/upgrade-migrations.html#_rollback_by_restoring_a_backup_snapshot

Can you try to follow the instructions again? Otherwise to proceed you'll can delete .kibana_task_manager_2 to release the lock and enable Kibana to try again.

3

Thanks for the suggestions. Following the upgrade guide once again did not solve it. Deleting .kibana_task* did however. So some task management state has been lost, but the rest is all looking good. So, solved for me thanks to your last suggestion. Much appreciated.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.