Kibana upgrade failed[8 -> 8.2.2]

Elastic Stack Kibana
1

Hi,

I tried to migrate from 8 to 8.2.2 today on a dev machine. This is having lot of spaces and objects.
As soon as I attempt, I landed into issues. The kibana migration has failed in mid.
I have tried to search the documentation. but in my case, unable to find the object(s) which is causing the issue.

kibana log:
[2022-05-28T15:36:00.360+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_READ. took: 108ms.
[2022-05-28T15:36:00.415+00:00][INFO ][savedobjects-service] [.kibana] Processed 20000 documents out of 20639.
[2022-05-28T15:36:00.415+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_TRANSFORM. took: 55ms.
[2022-05-28T15:36:00.699+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_READ. took: 284ms.
[2022-05-28T15:36:00.706+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> FATAL. took: 7ms.
[2022-05-28T15:36:00.710+00:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index: Migrations failed. Reason: 1 transformation errors were encountered:

  • canvas-workpad:workpad-2d8ea702-c9ca-4013-8294-504fd583dc21: Error: Migration function for version 8.1.0 threw an error
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:603:13)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:665:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:389:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:329:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:138:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:152:17)
    at processImmediate (node:internal/timers:466:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    Caused by:
    TypeError: Cannot read properties of undefined (reading 'state')
    at commonRenameFilterReferences (/usr/share/kibana/x-pack/plugins/lens/server/migrations/common_migrations.js:172:38)
    at Object.8.1.0 (/usr/share/kibana/x-pack/plugins/lens/server/embeddable/make_lens_embeddable_factory.js:58:136)
    at migrateFn (/usr/share/kibana/src/plugins/embeddable/common/lib/migrate.js:25:48)
    at migrations. (/usr/share/kibana/src/plugins/embeddable/common/lib/get_all_migrations.js:38:42)
    at Object.8.1.0 (/usr/share/kibana/x-pack/plugins/canvas/canvas_plugin_src/functions/external/embeddable.js:47:29)
    at /usr/share/kibana/src/plugins/expressions/common/executor/executor.js:322:33
    at /usr/share/kibana/src/plugins/expressions/common/executor/executor.js:234:29
    at Array.reduce ()
    at Executor.walkAstAndTransform (/usr/share/kibana/src/plugins/expressions/common/executor/executor.js:220:43)
    at Executor.migrate (/usr/share/kibana/src/plugins/expressions/common/executor/executor.js:315:17)
    at migrations. (/usr/share/kibana/src/plugins/expressions/common/executor/executor.js:304:49)
    at /usr/share/kibana/node_modules/lodash/lodash.js:5180:35
    at migrateExpr (/usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:23:91)
    at /usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:31:28
    at Array.map ()
    at /usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:53:86
    at Array.map ()
    at migrateWorkpadElements (/usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:49:72)
    at /usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:67:63
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:590:22)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:665:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:389:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:329:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:138:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:152:17)
    at processImmediate (node:internal/timers:466:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)

To allow migrations to proceed, please delete or fix these documents.
at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:144:29)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 0)
at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:213:9)
at Server.start (/usr/share/kibana/src/core/server/server.js:329:31)
at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)
at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)
at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)
[2022-05-28T15:36:00.716+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2022-05-28T15:36:00.717+00:00][INFO ][plugins-system.standard] Stopping all plugins.
[2022-05-28T15:36:00.717+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped
[2022-05-28T15:36:30.723+00:00][WARN ][plugins-system.standard] "eventLog" plugin didn't stop in 30sec., move on to the next.

FATAL Error: Unable to complete saved object migrations for the [.kibana] index: Migrations failed. Reason: 1 transformation errors were encountered:

  • canvas-workpad:workpad-2d8ea702-c9ca-4013-8294-504fd583dc21: Error: Migration function for version 8.1.0 threw an error
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:603:13)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:665:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:389:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:329:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:138:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:152:17)
    at processImmediate (node:internal/timers:466:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    Caused by:
    TypeError: Cannot read properties of undefined (reading 'state')
    at commonRenameFilterReferences (/usr/share/kibana/x-pack/plugins/lens/server/migrations/common_migrations.js:172:38)
    at Object.8.1.0 (/usr/share/kibana/x-pack/plugins/lens/server/embeddable/make_lens_embeddable_factory.js:58:136)
    at migrateFn (/usr/share/kibana/src/plugins/embeddable/common/lib/migrate.js:25:48)
    at migrations. (/usr/share/kibana/src/plugins/embeddable/common/lib/get_all_migrations.js:38:42)
    at Object.8.1.0 (/usr/share/kibana/x-pack/plugins/canvas/canvas_plugin_src/functions/external/embeddable.js:47:29)
    at /usr/share/kibana/src/plugins/expressions/common/executor/executor.js:322:33
    at /usr/share/kibana/src/plugins/expressions/common/executor/executor.js:234:29
    at Array.reduce ()
    at Executor.walkAstAndTransform (/usr/share/kibana/src/plugins/expressions/common/executor/executor.js:220:43)
    at Executor.migrate (/usr/share/kibana/src/plugins/expressions/common/executor/executor.js:315:17)
    at migrations. (/usr/share/kibana/src/plugins/expressions/common/executor/executor.js:304:49)
    at /usr/share/kibana/node_modules/lodash/lodash.js:5180:35
    at migrateExpr (/usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:23:91)
    at /usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:31:28
    at Array.map ()
    at /usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:53:86
    at Array.map ()
    at migrateWorkpadElements (/usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:49:72)
    at /usr/share/kibana/x-pack/plugins/canvas/server/saved_objects/migrations/expressions.js:67:63
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:590:22)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:665:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:389:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:329:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:138:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:152:17)
    at processImmediate (node:internal/timers:466:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)

To allow migrations to proceed, please delete or fix these documents.

$

I have set the aliases of .kibana and .kibana_task_manager to use the previous version of kibana as of now. Can some one help me in resolving the migration errors.

Thanks

2

Removed canvas object and re-attempt of upgrade failed with "Request timed out"

[2022-05-30T16:20:07.845+00:00][INFO ][savedobjects-service] [.kibana] Processed 20429 documents out of 20429.
[2022-05-30T16:20:07.845+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_READ -> REINDEX_SOURCE_TO_TEMP_CLOSE_PIT. took: 7ms.
[2022-05-30T16:20:07.849+00:00][INFO ][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_CLOSE_PIT -> SET_TEMP_WRITE_BLOCK. took: 4ms.
[2022-05-30T16:20:07.949+00:00][INFO ][savedobjects-service] [.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 100ms.
[2022-05-30T16:20:37.954+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds.
[2022-05-30T16:20:37.955+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 30004ms.
[2022-05-30T16:22:39.975+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 2 in 4 seconds.
[2022-05-30T16:22:39.975+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 122022ms.
[2022-05-30T16:24:43.991+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 3 in 8 seconds.
[2022-05-30T16:24:43.991+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 124016ms.
[2022-05-30T16:26:52.019+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 4 in 16 seconds.
[2022-05-30T16:26:52.019+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 128028ms.
[2022-05-30T16:29:08.033+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 5 in 32 seconds.
[2022-05-30T16:29:08.034+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 136014ms.
[2022-05-30T16:31:40.052+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 6 in 64 seconds.
[2022-05-30T16:31:40.053+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 152019ms.
[2022-05-30T16:34:44.064+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 7 in 64 seconds.
[2022-05-30T16:34:44.065+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184012ms.
[2022-05-30T16:37:48.081+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 8 in 64 seconds.
[2022-05-30T16:37:48.083+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184017ms.
[2022-05-30T16:39:57.440+00:00][ERROR][plugins.ruleRegistry] Error: Timeout: it took more than 1200000ms
at Timeout._onTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:54:20)
at listOnTimeout (node:internal/timers:559:17)
at processTimers (node:internal/timers:502:7)
[2022-05-30T16:39:57.442+00:00][ERROR][plugins.ruleRegistry] Error: Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms
at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:63:13)
at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:77:5)
[2022-05-30T16:40:52.101+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 9 in 64 seconds.
[2022-05-30T16:40:52.102+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184020ms.
[2022-05-30T16:43:56.115+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 10 in 64 seconds.
[2022-05-30T16:43:56.115+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184014ms.
[2022-05-30T16:47:00.145+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 11 in 64 seconds.
[2022-05-30T16:47:00.145+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184030ms.
[2022-05-30T16:50:04.159+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 12 in 64 seconds.
[2022-05-30T16:50:04.160+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184014ms.
[2022-05-30T16:53:08.176+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 13 in 64 seconds.
[2022-05-30T16:53:08.176+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184017ms.
[2022-05-30T16:56:12.193+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 14 in 64 seconds.
[2022-05-30T16:56:12.194+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184017ms.
[2022-05-30T16:59:16.202+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Request timed out'. Retrying attempt 15 in 64 seconds.
[2022-05-30T16:59:16.203+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 184009ms.
[2022-05-30T17:02:20.219+00:00][INFO ][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> FATAL. took: 184017ms.
[2022-05-30T17:02:20.220+00:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the CLONE_TEMP_TO_TARGET step after 15 attempts, terminating.
at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:144:29)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Promise.all (index 0)
at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:213:9)
at Server.start (/usr/share/kibana/src/core/server/server.js:329:31)
at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)
at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)
at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)
[2022-05-30T17:02:20.229+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2022-05-30T17:02:20.230+00:00][INFO ][plugins-system.standard] Stopping all plugins.
[2022-05-30T17:02:20.230+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped
[2022-05-30T17:02:50.235+00:00][WARN ][plugins-system.standard] "eventLog" plugin

3

Found the .kibana_8.2.2 index in red status with kibana 8(previous version). Deleted it and made one more attempt.
This time it is successful.

4

I am getting almost the exact same errors between a 8.1 to 8.2.1 upgrade. Kibana is failing to start. How did you remove the canvas objects?

5
  1. I have restored the alias to previous and taken backup of objects as ndjson.
  2. Searched and deleted canvas
  3. Deleted new kibana in my case.
  4. Upgraded again. This time, it worked.
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.