Kibana won't start after 8.2.0 upgrade

I had 8.1.1 running and I upgraded to 8.2.0

this is single node test system. hence I shutdown Elasticsearch and kibana
upgrade and start

kibana is complaining like this

[2022-05-16T19:33:11.618-05:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_8.1.1_001" index.
The documents with unknown types are:
- "telemetry:telemetry" (type: "telemetry")
You can delete them using the following command:
curl -X POST "{elasticsearch}/.kibana_8.1.1_001/_bulk?pretty" -H 'Content-Type: application/json' -d'
{ "delete" : { "_id" : "telemetry:telemetry" } }
'
    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:144:29)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:213:9)
    at Server.start (/usr/share/kibana/src/core/server/server.js:329:31)
    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)
    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)
[2022-05-16T19:33:11.630-05:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2022-05-16T19:33:11.634-05:00][INFO ][plugins-system.standard] Stopping all plugins.
[2022-05-16T19:33:11.637-05:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped

then I try to execute this curl command which is telling me to delete telemetry. but it fails


{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices

I even try deleting some old index but I can't do that anymore


# curl -u elastic:elastic -XDELETE houelkv8:9200/.kibana_task_manager_7.16.2_001
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}

and user:password is correct as I can see the cluster status

# curl -u elastic:elastic -XGET houelkv8:9200/_cluster/health?pretty
{
  "cluster_name" : "elk_v_8_0",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 48,
  "active_shards" : 48,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

How do I fix it?

I don't even see .kibana_8.1.1_001 when I list _cat/indices

anyone has such issue?

Hi @elasticforme

Check this thread same issue as yours.

Hello,

In order to help, I would like to ask if you can just start Elasticsearch and then check via curl, as elastic user:

GET _cat/indices/.kibana*?expand_wildcards=all&v

You should be able to see the index .kibana_8.1.1_001.
The index .kibana_8.2.0_001 might exist, but it will have less documents than 8.1.1.

The output of:

GET _cat/aliases/.kibana*?expand_wildcards=all&v

Should instead show .kibana pointing still to 8.1.1 indices.

In order to edit system indices in 8+, you need a special permission.
It is documented here.

Create a role as follows:

PUT _security/role/grant_kibana_indices
{
  "indices": [
    {
      "names": [
        ".kibana*"
      ],
      "privileges": [
        "all"
      ],
      "allow_restricted_indices": true
    }
  ]
}

Create a user with the role above and superuser built-in role:

POST /_security/user/temporarykibanasuperuser
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : [ "superuser", "grant_kibana_indices" ]
}

Run:

PUT .kibana_8.1.1_001/_settings
{
  "index": {
    "blocks.write": false
  }
}

Run (modify {elasticsearch} with the correct host), you will be prompted for a password:

curl -u temporarykibanasuperuser -X POST "{elasticsearch}/.kibana_8.1.1_001/_bulk?pretty" -H 'Content-Type: application/json' -d'{ "delete" : { "_id" : "telemetry:telemetry" } }'

Once done, start Kibana 8.2.0 and it should start and be able to finalize the Saved Objects migration.

To be honest, I've not seen other issues like this in the Kibana repository, so if you're able to reproduce this problem consistently, I would suggest filing an issue on the Kibana repository.


The procedure on how to solve Saved Objects issues is at this page, but it doesn't mention the need of creating a special permission.
I've opened [DOCS] Explain how to create a special user to solve Saved Object conflicts (edit Kibana index) · Issue #132370 · elastic/kibana · GitHub

1 Like

will this be fixed in next version?

I think I can produce this. if I redo 8.1.1 install and then upgrade again.

@elasticforme The upgrade migrations fail because it seems like you disabled the telemetry plugin which results in the plugin not being loaded at all. As a result, Kibana doesn't know how to handle the data that belonged that plugin. The recommended way to disable telemetry is using telemetry.optIn: false Deprecate `telemetry.enabled: false` · Issue #129015 · elastic/kibana · GitHub The telemetry plugin will still be loaded, but it won't transmit any telemetry back to Elastic.

Now that you've removed the unknown saved object further upgrades won't run into this problem again so this won't affect you when e.g. upgrading to 8.2.0.

We also hope to make it easier to resolve upgrade failures like this by adding a new configuration flag Add migrations.ignoreUnknownObjects flag by gsoldevila · Pull Request #132167 · elastic/kibana · GitHub

1 Like

Thank you Rudolf. this makes complete sense.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.