Kibana 7.5.2 to 7.17.0 Migration Failure - Migration of Saved Objects Failure

[anburethy]

Hi, I am migrating Kibana from 7.5.2 to 7.17.0 and getting the following error when doing so. Kibana isnt starting up due to this failure and hence migration fails.

{"type":"log","@timestamp":"2022-03-20T12:32:03+00:00","tags":["fatal","root"],"pid":76789,"message":"Error: Unable to complete saved object migrations for the [.kibana] index: Migrations failed. Reason: 1 transformation errors were encountered:\n - platform:dashboard:1234-530f-11ea-b8ea-sdf323232: Error: Migration function for version 7.11.0 threw an error\n    at tryTransformDoc (/usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:610:13)\n    at migrateProp (/usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)\n    at applyMigrations (/usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)\n    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)\n    at /usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16\n    at Immediate._onImmediate (/usr/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)\n    at processImmediate (node:internal/timers:464:21)
\nCaused by:\nError: Could not find reference \"panel_0\"\n    at Object.inject (/usr/kibana/src/plugins/dashboard/common/embeddable/dashboard_container_persistable_state.js:40:19)\n    at Object.inject (/usr/kibana/src/plugins/embeddable/common/lib/inject.js:24:30)\n    at injectReferences (/usr/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:134:64)\n    at /usr/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:142:81\n    at /usr/kibana/node_modules/lodash/lodash.js:5177:46\n    at tryTransformDoc (/usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)\n    at migrateProp (/usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)\n    at applyMigrations (/usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)\n    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)\n    at /usr/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16\n    at Immediate._onImmediate (/usr/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)\n    at processImmediate (node:internal/timers:464:21)\n\nTo allow migrations to proceed, please delete or fix these documents.\n    at migrationStateActionMachine (/usr/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:144:29)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 0)\n
    at SavedObjectsService.start (/usr/kibana/src/core/server/saved_objects/saved_objects_service.js:181:9)\n    at Server.start (/usr/kibana/src/core/server/server.js:330:31)\n    at Root.start (/usr/kibana/src/core/server/root/index.js:69:14)\n    at bootstrap (/usr/kibana/src/core/server/bootstrap.js:120:5)\n    at Command.<anonymous> (/usr/kibana/src/cli/serve/serve.js:229:5)"}

[ropc (Romain Chanu)]

Hi @anburethy , based on the error message, the saved object platform:dashboard:1234-530f-11ea-b8ea-sdf323232 cannot be migrated. It could be possible that this object is corrupt.

If so, you could delete this object from the .kibana index (i.e similar steps given in Corrupt saved objects).

Can you try this approach?

[anburethy]

Thanks @ropc , I can delete the object but my Kibana is under migration and its not starting up now and I can revert as well now. So I wont be able to run a command in Kibana UI or curl. Any help on how to find and delete it manually from the server.

[ropc (Romain Chanu)]

@anburethy - if the Elasticsearch cluster is running and is reachable, you should be able to run the required APIs (e.g Delete API) via curl, Postman.