Kibana 6.8 to 7.3 fails: [uiStateJSON] within [dashboard] is not allowed

Elastic Stack Kibana
1

Just tried to upgrade Kibana from a working 6.8 to 7.3 and kibana will not start saying:

FATAL Error: mapping set to strict, dynamic introduction of [uiStateJSON] within [dashboard] is not allowed

log   [11:51:53.497] [info][status][plugin:reporting@7.3.0] Status changed from uninitialized to green - Ready
      log   [11:51:53.699] [warning][task_manager] Task vis_telemetry "oss_telemetry-vis_telemetry" failed in attempt to run: [version_conflict_engine_exception] [oss_telemetry-vis_telemetry]: version conflict, required seqNo [112], primary term [5]. current document has seqNo [113] and primary term [5], with { index_uuid="hm7O8_iTSleeVAHlYpORuA" & shard="0" & index=".kibana_task_manager" }
      log   [11:51:53.714] [info][migrations] Creating index .kibana_9.
      log   [11:51:53.901] [info][migrations] Migrating .kibana_8 saved objects to .kibana_9
      log   [11:51:53.947] [warning][migrations] Exception @ migrations730 while trying to migrate dashboard panels! TypeError: Cannot set property 'columns' of undefined
      log   [11:51:54.410] [error][status][plugin:spaces@7.3.0] Status changed from yellow to red - mapping set to strict, dynamic introduction of [uiStateJSON] within [dashboard] is not allowed
      log   [11:51:54.414] [fatal][root] { Error: mapping set to strict, dynamic introduction of [uiStateJSON] within [dashboard] is not allowed
        at Object.write (D:\elk\kibana-7.3.0\src\core\server\saved_objects\migrations\core\elastic_index.js:152:21)
        at process._tickCallback (internal/process/next_tick.js:68:7)
      detail:
       { index:
          { _index: '.kibana_9',
            _type: '_doc',
            _id: 'dashboard:Monitoring-Status-(TEST)',
            status: 400,
            error: [Object] } },
      isBoom: true,
      isServer: true,
      data: null,
      output:
       { statusCode: 500,
         payload:
          { statusCode: 500,
            error: 'Internal Server Error',
            message: 'An internal server error occurred' },
         headers: {} },
      reformat: [Function],
      [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/generalError' }
      log   [11:51:54.421] [info][plugins-system] Stopping all plugins.
      log   [11:51:54.422] [info][plugins][translations] Stopping plugin

     FATAL  Error: mapping set to strict, dynamic introduction of [uiStateJSON] within [dashboard] is not allowed

Here is part of our kibana index where we can indeed see uiStateJSON inside dashboard:

        {
            "_index": ".kibana_8",
            "_type": "doc",
            "_id": "dashboard:5f08a870-7c6a-11e7-aa55-3b0d52c71c60",
            "_score": 1.0,
            "_source": {
                "dashboard": {
                    "title": "[APM] Error Details",
                    "hits": 0,
                    "description": "",
                    "panelsJSON": "[...]",
                    "optionsJSON": "{\"darkTheme\": false}",
                    "uiStateJSON": "{}",
2

Solution was to:

  • run the old Kibana and
  • export all dashboards from Management -> Saved Objects.
  • Edit this JSON file and remove all uiStateJSON elements from the JSON.
  • Save and re-upload the JSON with overwrite

Now you can stop kibana and upgrade.

Note: if an upgrade failed you may have a kibana_N index where N is higher than your current .kibana alias. This index can be deleted.

Example:
GET /_cat/aliases/kib -> Reveals kibana is using the .kibana_8
However, the failed migration created a .kibana_9 index which can be deleted.