# Kibana won't start after 8.2.0 upgrade

**URL:** https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881
**Category:** Kibana
**Created:** [May 17, 2022, 12:42am UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881 "2022-05-17T00:42:14Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [May 17, 2022, 12:42am UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881/1 "2022-05-17T00:42:14Z")

</div>

I had 8.1.1 running and I upgraded to 8.2.0

this is single node test system. hence I shutdown Elasticsearch and kibana  
upgrade and start

kibana is complaining like this

```auto
[2022-05-16T19:33:11.618-05:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_8.1.1_001" index.
The documents with unknown types are:
- "telemetry:telemetry" (type: "telemetry")
You can delete them using the following command:
curl -X POST "{elasticsearch}/.kibana_8.1.1_001/_bulk?pretty" -H 'Content-Type: application/json' -d'
{ "delete" : { "_id" : "telemetry:telemetry" } }
'
    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:144:29)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:213:9)
    at Server.start (/usr/share/kibana/src/core/server/server.js:329:31)
    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)
    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)
[2022-05-16T19:33:11.630-05:00][INFO][plugins-system.preboot] Stopping all plugins.
[2022-05-16T19:33:11.634-05:00][INFO][plugins-system.standard] Stopping all plugins.
[2022-05-16T19:33:11.637-05:00][INFO][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped

```

then I try to execute this curl command which is telling me to delete telemetry. but it fails

```auto

{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices

```

I even try deleting some old index but I can't do that anymore

```auto

# curl -u elastic:elastic -XDELETE houelkv8:9200/.kibana_task_manager_7.16.2_001
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}

```

and user:password is correct as I can see the cluster status

```auto
# curl -u elastic:elastic -XGET houelkv8:9200/_cluster/health?pretty
{
  "cluster_name" : "elk_v_8_0",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 48,
  "active_shards" : 48,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

```

How do I fix it?

---

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [May 17, 2022, 4:58pm UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881/2 "2022-05-17T16:58:19Z")

</div>

I don't even see .kibana\_8.1.1\_001 when I list \_cat/indices

anyone has such issue?

---

<div class="post-metadata">

### Author: ![ibra\_013](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibra_013/32/104827_2.png) [@ibra\_013](https://discuss.elastic.co/u/ibra_013)
#### Post date: [May 17, 2022, 8:24pm UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881/3 "2022-05-17T20:24:16Z")

</div>

Hi @elasticforme

Check this [thread](https://discuss.elastic.co/t/cannot-delete-index-with-superuser/303249/3) same issue as yours.

> <https://github.com/elastic/kibana/issues/127881>
>
> \*\*Kibana version:\*\* 8.0+
> 
> \*\*Elasticsearch version:\*\* aligned with Kibana
> 
> \*\*…Describe the bug:\*\*
> 
> On versions \< 8.0, \`.reporting\*\` indices could be deleted at least by the \`superuser\` or anyone who had \`reporting\_user\` permissions.
> 
> On 8.0+, due to the fact \`.reporting\*\` indices are considered system indices, even a \`superuser\` do not have the permission to delete them (as \`superuser\` doesn't have the permission to delete restricted indices starting 8.0).
> 
> \*\*Steps to reproduce:\*\*
> 1. Create report
> 2. Try to delete any \`.reporting-\*\` index
> 
> \*\*Expected behavior:\*\*
> 
> A user should have the permission to delete reports and/or clean up old reports without having to use workarounds.
> 
> The current workaround is to create a dedicated role and a user assigned to it which grants:
> \`\`\`
> {
> "names" : \[
> ".reporting\*"
> \],
> "privileges" : \[
> "monitor",
> "read",
> "view\_index\_metadata",
> "read\_cross\_cluster",
> "delete\_index"
> \],
> "allow\_restricted\_indices" : true
> }
> \`\`\`
> 
> The problem occurs also when granting both \`reporting\_user\` and \`superuser\` permissions.
> \`\`\`
> {
> "error" : {
> "root\_cause" : \[
> {
> "type" : "security\_exception",
> "reason" : "action \[indices:admin/delete\] is unauthorized for user \[lucalucaluca\] with roles \[reporting\_user,superuser\] on indices \[.reporting-2022-03-13\], this action is granted by the index privileges \[delete\_index,manage,all\]"
> }
> \],
> "type" : "security\_exception",
> "reason" : "action \[indices:admin/delete\] is unauthorized for user \[lucalucaluca\] with roles \[reporting\_user,superuser\] on indices \[.reporting-2022-03-13\], this action is granted by the index privileges \[delete\_index,manage,all\]"
> },
> "status" : 403
> }
> \`\`\`
> 
> The same problem occurs when using the new Kibana permissions.
> Even granting the following to a given user, still blocks the removal of \`.reporting\*\` indices.
> \`\`\`
> {
> "reporting\_my" : {
> "cluster" : \[
> "all"
> \],
> "indices" : \[
> {
> "names" : \[
> "kibana\_sample\_data\_logs"
> \],
> "privileges" : \[
> "all"
> \],
> "field\_security" : {
> "grant" : \[
> "\*"
> \],
> "except" : \[\]
> },
> "allow\_restricted\_indices" : false
> }
> \],
> "applications" : \[
> {
> "application" : "kibana-.kibana",
> "privileges" : \[
> "feature\_discover.minimal\_all",
> "feature\_discover.url\_create",
> "feature\_discover.store\_search\_session",
> "feature\_discover.generate\_report",
> "feature\_dashboard.minimal\_all",
> "feature\_dashboard.url\_create",
> "feature\_dashboard.store\_search\_session",
> "feature\_dashboard.generate\_report",
> "feature\_dashboard.download\_csv\_report"
> \],
> "resources" : \[
> "\*"
> \]
> }
> \],
> "run\_as" : \[\],
> "metadata" : { },
> "transient\_metadata" : {
> "enabled" : true
> }
> }
> }
> 
> \`\`\`
> 
> \*\*Any additional context:\*\*
> 
> As \`.reporting\*\` documents contains both the job/task definition and the "result" of the report, maybe we could decouple them into 2 separate indices? Also the \`.reporting\` index could become a "workspace" index (like \`.async-search\` index) where we keep only the jobs to be done and store the generated reports in a separate index.
> 
> Or otherwise we should provide a dedicated API to manage reports \`\_reporting\` at ES or Kibana level and allow to delete specific reports or whole groups of reports.
> 
> ~I would consider it a bug as it is a regression on features of reporting~
> \> This as an effect of the breaking change made by ES security to no longer allow the superuser role to manage system indices by default.
> 
> \*\*PLEASE BE AWARE\*\* that it is possible to auto-delete old Reporting indices modifying the ILM Policy \`kibana-reporting\`, adding a Delete phase.
> So, this problem affects only users who are interested in deleting a specific reporting index.

---

<div class="post-metadata">

### Author: ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)
#### Post date: [May 17, 2022, 8:57pm UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881/4 "2022-05-17T20:57:18Z")

</div>

Hello,

In order to help, I would like to ask if you can just start Elasticsearch and then check via `curl`, as `elastic` user:

```auto
GET _cat/indices/.kibana*?expand_wildcards=all&v

```

You should be able to see the index `.kibana_8.1.1_001`.  
The index `.kibana_8.2.0_001` might exist, but it will have less documents than 8.1.1.

The output of:

```auto
GET _cat/aliases/.kibana*?expand_wildcards=all&v

```

Should instead show `.kibana` pointing still to 8.1.1 indices.

In order to edit system indices in 8+, you need a special permission.  
It is documented [here](https://www.elastic.co/guide/en/elasticsearch/reference/master/migrating-8.0.html#breaking_80_index_setting_changes).

Create a role as follows:

```auto
PUT _security/role/grant_kibana_indices
{
  "indices": [
    {
      "names": [
        ".kibana*"
      ],
      "privileges": [
        "all"
      ],
      "allow_restricted_indices": true
    }
  ]
}

```

Create a user with the role above and `superuser` built-in role:

```auto
POST /_security/user/temporarykibanasuperuser
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : ["superuser", "grant_kibana_indices"]
}

```

Run:

```auto
PUT .kibana_8.1.1_001/_settings
{
  "index": {
    "blocks.write": false
  }
}

```

Run (modify `{elasticsearch}` with the correct host), you will be prompted for a password:

```auto
curl -u temporarykibanasuperuser -X POST "{elasticsearch}/.kibana_8.1.1_001/_bulk?pretty" -H 'Content-Type: application/json' -d'{ "delete" : { "_id" : "telemetry:telemetry" } }'

```

Once done, start Kibana 8.2.0 and it should start and be able to finalize the Saved Objects migration.

To be honest, I've not seen other issues like this in the Kibana repository, so if you're able to reproduce this problem consistently, I would suggest filing an issue on the Kibana repository.

* * *

The procedure on how to solve Saved Objects issues is [at this page](https://www.elastic.co/guide/en/kibana/8.2/resolve-migrations-failures.html#_corrupt_saved_objects), but it doesn't mention the need of creating a special permission.  
I've opened [[DOCS] Explain how to create a special user to solve Saved Object conflicts (edit Kibana index) · Issue #132370 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/132370)

---

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [May 17, 2022, 9:29pm UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881/5 "2022-05-17T21:29:52Z")

</div>

will this be fixed in next version?

I think I can produce this. if I redo 8.1.1 install and then upgrade again.

---

<div class="post-metadata">

### Author: ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)
#### Post date: [May 18, 2022, 10:08am UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881/6 "2022-05-18T10:08:49Z")

</div>

@elasticforme The upgrade migrations fail because it seems like you disabled the telemetry plugin which results in the plugin not being loaded at all. As a result, Kibana doesn't know how to handle the data that belonged that plugin. The recommended way to disable telemetry is using `telemetry.optIn: false` [Deprecate `telemetry.enabled: false` · Issue #129015 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/129015) The telemetry plugin will still be loaded, but it won't transmit any telemetry back to Elastic.

Now that you've removed the unknown saved object further upgrades won't run into this problem again so this won't affect you when e.g. upgrading to 8.2.0.

We also hope to make it easier to resolve upgrade failures like this by adding a new configuration flag [Add migrations.ignoreUnknownObjects flag by gsoldevila · Pull Request #132167 · elastic/kibana · GitHub](https://github.com/elastic/kibana/pull/132167)

---

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [May 18, 2022, 1:52pm UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881/7 "2022-05-18T13:52:04Z")

</div>

> [@rudolf](#):
>
> `telemetry.optIn: false`

Thank you Rudolf. this makes complete sense.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 15, 2022, 1:52pm UTC](https://discuss.elastic.co/t/kibana-wont-start-after-8-2-0-upgrade/304881/8 "2022-06-15T13:52:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
