Kibana: Error during migration from 7.10.0 to 7.16.1

obi134 · December 17, 2021, 12:07pm

Hi there,
We ran an update of ELK via yum. Previously we had 7.10.0 and now updated to 7.16.1.

Now we have problems starting kibana. we got the following error message:

 FATAL  Error: Unable to complete saved object migrations for the [.kibana] index: Migrations failed. Reason: 2 transformation errors were encountered:
 - dashboard:7100b3f0-50bb-11eb-8710-957440b45d79: Error: Migration function for version 7.11.0 threw an error
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:610:13)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)
    at processImmediate (node:internal/timers:464:21)
Caused by:
TypeError: Invalid Version:
    at new SemVer (/usr/share/kibana/node_modules/semver/classes/semver.js:41:13)
    at compare (/usr/share/kibana/node_modules/semver/functions/compare.js:3:32)
    at gt (/usr/share/kibana/node_modules/semver/functions/gt.js:2:29)
    at isPre730Panel (/usr/share/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:23:47)
    at Array.some (<anonymous>)
    at extractReferences (/usr/share/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:87:14)
    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:151:59
    at /usr/share/kibana/node_modules/lodash/lodash.js:5177:46
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)
    at processImmediate (node:internal/timers:464:21)
- dashboard:f77473d0-50bc-11eb-8710-957440b45d79: Error: Migration function for version 7.11.0 threw an error
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:610:13)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)
    at processImmediate (node:internal/timers:464:21)
Caused by:
TypeError: Invalid Version:
    at new SemVer (/usr/share/kibana/node_modules/semver/classes/semver.js:41:13)
    at compare (/usr/share/kibana/node_modules/semver/functions/compare.js:3:32)
    at gt (/usr/share/kibana/node_modules/semver/functions/gt.js:2:29)
    at isPre730Panel (/usr/share/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:23:47)
    at Array.some (<anonymous>)
    at extractReferences (/usr/share/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:87:14)
    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:151:59
    at /usr/share/kibana/node_modules/lodash/lodash.js:5177:46
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)
    at processImmediate (node:internal/timers:464:21)

To allow migrations to proceed, please delete or fix these documents.

I used curl -XGET localhost:9200/_cat/aliases and got .kibana .kibana_1

After this I used curl -XGET localhost:9200/_cat/indices/.kibana_1 and got green open .kibana_1 6oWws2XgSZW3BJn9u2WbYw 1 0 1790 7610 11.4mb 11.4mb

How to get those both documents (7100b3f0-50bb-11eb-8710-957440b45d79, f77473d0-50bc-11eb-8710-957440b45d79), see, what is inside and maybe delete them by using curl?
Or is there another way to get around this?

obi134 · December 17, 2021, 1:03pm

Update:
I could now read it with:

curl -X GET "localhost:9200/.kibana_1/_doc/_search?pretty=true" -H 'Content-Type: application/json' -d'
{
    "query" : {
        "match": {
      "_id": "dashboard:f77473d0-50bc-11eb-8710-957440b45d79"
    }
    }
}
'

Now I'm wondering if I can fix this document. It has following structure:

"hits" : [
      {
        "_index" : ".kibana_1",
        "_type" : "_doc",
        "_id" : "dashboard:f77473d0-50bc-11eb-8710-957440b45d79",
        "_score" : 1.0,
        "_source" : {
          "dashboard" : {
            "description" : "",
            "hits" : 0,
            "kibanaSavedObjectMeta" : {
              "searchSourceJSON" : "{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"
            },
            "optionsJSON" : "{\"hidePanelTitles\":false,\"useMargins\":true}",
            "panelsJSON" : "[{\"embeddableConfig\": {\"hidePanelTitles\": true}, \"gridData\": {\"x\": 0, \"y\": 0, \"w\": 48, \"h\": 3, \"i\": \"0\"}, \"panelIndex\": \"0\", \"panelRefName\": \"panel_0\"}, {\"embeddableConfig\": {}, \"$
            "timeFrom" : "now-30d",
            "timeRestore" : true,
            "timeTo" : "now",
            "title" : "Overview Other Attributes",
            "version" : 1
          },
          "type" : "dashboard",
          "references" : [
            {
              "id" : "d5a96de0-5018-11eb-8710-957440b45d79",
              "name" : "panel_0",
              "type" : "visualization"
            },
            ...(a lot of more visualization)...
          {
              "id" : "862c3ef0-5004-11eb-8710-957440b45d79",
              "name" : "panel_137",
              "type" : "visualization"
            }
           ],
          "migrationVersion" : {
            "dashboard" : "7.9.3"
          },
          "updated_at" : "2021-01-07T07:50:10.060Z"
        }

tsullivan · December 17, 2021, 9:22pm

Hi, I believe you can fix this dashboard saved object document so that the migration will complete.

I could reproduce this problem after creating a dashboard in 7.9.3, exporting the saved objects using Stack Management > Saved Objects, and then manually editing the export.ndjson file to remove some of the information. This screenshot shows the information that I removed:

I would suggest getting an export.ndjson of the saved objects in 7.10, manually edit the export file in an editor to add a version field for each item in panelsJSON, and import the edited file into 7.16.1.

obi134 · December 20, 2021, 9:36am

Hi Tim,
Thank you for your answer. Could you please be more precise? Did you remove those version fields or did you add?
Above the picture you wrote "to remove some of the information" and below it you wrote "to add a version field"

Currently I don't have a version field in my panelsJSON

tsullivan · December 20, 2021, 4:38pm

Hiya - sorry about that you caught me with a typo.

The image shows data with fields that are highlighted - those fields are required. When I ran a test without those fields, I encountered the error that you are encountering. You need to add the field to each item in the panelsJSON.

obi134 · January 3, 2022, 10:04am

Thank you Tim,
I tried to insert the version fields but now I got internal server error when importing. Unfortunately the log is not giving any hint.
So I guess I need to create the dashboard again.
Thank you for your help!

system · January 31, 2022, 10:04am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
FATAL Error: Unable to complete saved object migrations for the [.kibana] index: Migrations failed. Reason: 2 transformation errors were encountered Kibana	1	200	July 17, 2023
Kibana 7.5.2 to 7.17.0 Migration Failure - Migration of Saved Objects Failure Kibana	4	822	April 18, 2022
Kibana update from 7.10.2 to 7.16.2 Kibana	2	542	January 25, 2022
Migration error Kibana	2	1271	February 7, 2022
Kibana error Kibana	2	266	August 2, 2022

Kibana: Error during migration from 7.10.0 to 7.16.1

Related topics