Kibana: Error during migration from 7.10.0 to 7.16.1

Elastic Stack Kibana
1

Hi there,
We ran an update of ELK via yum. Previously we had 7.10.0 and now updated to 7.16.1.

Now we have problems starting kibana. we got the following error message:

 FATAL  Error: Unable to complete saved object migrations for the [.kibana] index: Migrations failed. Reason: 2 transformation errors were encountered:
 - dashboard:7100b3f0-50bb-11eb-8710-957440b45d79: Error: Migration function for version 7.11.0 threw an error
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:610:13)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)
    at processImmediate (node:internal/timers:464:21)
Caused by:
TypeError: Invalid Version:
    at new SemVer (/usr/share/kibana/node_modules/semver/classes/semver.js:41:13)
    at compare (/usr/share/kibana/node_modules/semver/functions/compare.js:3:32)
    at gt (/usr/share/kibana/node_modules/semver/functions/gt.js:2:29)
    at isPre730Panel (/usr/share/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:23:47)
    at Array.some (<anonymous>)
    at extractReferences (/usr/share/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:87:14)
    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:151:59
    at /usr/share/kibana/node_modules/lodash/lodash.js:5177:46
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)
    at processImmediate (node:internal/timers:464:21)
- dashboard:f77473d0-50bc-11eb-8710-957440b45d79: Error: Migration function for version 7.11.0 threw an error
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:610:13)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)
    at processImmediate (node:internal/timers:464:21)
Caused by:
TypeError: Invalid Version:
    at new SemVer (/usr/share/kibana/node_modules/semver/classes/semver.js:41:13)
    at compare (/usr/share/kibana/node_modules/semver/functions/compare.js:3:32)
    at gt (/usr/share/kibana/node_modules/semver/functions/gt.js:2:29)
    at isPre730Panel (/usr/share/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:23:47)
    at Array.some (<anonymous>)
    at extractReferences (/usr/share/kibana/src/plugins/dashboard/common/saved_dashboard_references.js:87:14)
    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:151:59
    at /usr/share/kibana/node_modules/lodash/lodash.js:5177:46
    at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:597:22)
    at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:679:22)
    at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:397:20)
    at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:337:22)
    at /usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:146:16
    at Immediate._onImmediate (/usr/share/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)
    at processImmediate (node:internal/timers:464:21)

To allow migrations to proceed, please delete or fix these documents.

I used curl -XGET localhost:9200/_cat/aliases and got .kibana .kibana_1

After this I used curl -XGET localhost:9200/_cat/indices/.kibana_1 and got green open .kibana_1 6oWws2XgSZW3BJn9u2WbYw 1 0 1790 7610 11.4mb 11.4mb

How to get those both documents (7100b3f0-50bb-11eb-8710-957440b45d79, f77473d0-50bc-11eb-8710-957440b45d79), see, what is inside and maybe delete them by using curl?
Or is there another way to get around this?

2

Update:
I could now read it with:

curl -X GET "localhost:9200/.kibana_1/_doc/_search?pretty=true" -H 'Content-Type: application/json' -d'
{
    "query" : {
        "match": {
      "_id": "dashboard:f77473d0-50bc-11eb-8710-957440b45d79"
    }
    }
}
'

Now I'm wondering if I can fix this document. It has following structure:

"hits" : [
      {
        "_index" : ".kibana_1",
        "_type" : "_doc",
        "_id" : "dashboard:f77473d0-50bc-11eb-8710-957440b45d79",
        "_score" : 1.0,
        "_source" : {
          "dashboard" : {
            "description" : "",
            "hits" : 0,
            "kibanaSavedObjectMeta" : {
              "searchSourceJSON" : "{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"
            },
            "optionsJSON" : "{\"hidePanelTitles\":false,\"useMargins\":true}",
            "panelsJSON" : "[{\"embeddableConfig\": {\"hidePanelTitles\": true}, \"gridData\": {\"x\": 0, \"y\": 0, \"w\": 48, \"h\": 3, \"i\": \"0\"}, \"panelIndex\": \"0\", \"panelRefName\": \"panel_0\"}, {\"embeddableConfig\": {}, \"$
            "timeFrom" : "now-30d",
            "timeRestore" : true,
            "timeTo" : "now",
            "title" : "Overview Other Attributes",
            "version" : 1
          },
          "type" : "dashboard",
          "references" : [
            {
              "id" : "d5a96de0-5018-11eb-8710-957440b45d79",
              "name" : "panel_0",
              "type" : "visualization"
            },
            ...(a lot of more visualization)...
          {
              "id" : "862c3ef0-5004-11eb-8710-957440b45d79",
              "name" : "panel_137",
              "type" : "visualization"
            }
           ],
          "migrationVersion" : {
            "dashboard" : "7.9.3"
          },
          "updated_at" : "2021-01-07T07:50:10.060Z"
        }
3

Hi, I believe you can fix this dashboard saved object document so that the migration will complete.

I could reproduce this problem after creating a dashboard in 7.9.3, exporting the saved objects using Stack Management > Saved Objects, and then manually editing the export.ndjson file to remove some of the information. This screenshot shows the information that I removed:

image
image2485×1038 411 KB

I would suggest getting an export.ndjson of the saved objects in 7.10, manually edit the export file in an editor to add a version field for each item in panelsJSON, and import the edited file into 7.16.1.

4

Hi Tim,
Thank you for your answer. Could you please be more precise? Did you remove those version fields or did you add?
Above the picture you wrote "to remove some of the information" and below it you wrote "to add a version field"

Currently I don't have a version field in my panelsJSON

5

Hiya - sorry about that you caught me with a typo.

The image shows data with fields that are highlighted - those fields are required. When I ran a test without those fields, I encountered the error that you are encountering. You need to add the field to each item in the panelsJSON.

6

Thank you Tim,
I tried to insert the version fields but now I got internal server error when importing. Unfortunately the log is not giving any hint.
So I guess I need to create the dashboard again.
Thank you for your help!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.