Kibana won't start after 8.2.0 upgrade

elasticforme · May 17, 2022, 12:42am

I had 8.1.1 running and I upgraded to 8.2.0

this is single node test system. hence I shutdown Elasticsearch and kibana
upgrade and start

kibana is complaining like this

[2022-05-16T19:33:11.618-05:00][FATAL][root] Error: Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_8.1.1_001" index.
The documents with unknown types are:
- "telemetry:telemetry" (type: "telemetry")
You can delete them using the following command:
curl -X POST "{elasticsearch}/.kibana_8.1.1_001/_bulk?pretty" -H 'Content-Type: application/json' -d'
{ "delete" : { "_id" : "telemetry:telemetry" } }
'
    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:144:29)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:213:9)
    at Server.start (/usr/share/kibana/src/core/server/server.js:329:31)
    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)
    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)
[2022-05-16T19:33:11.630-05:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2022-05-16T19:33:11.634-05:00][INFO ][plugins-system.standard] Stopping all plugins.
[2022-05-16T19:33:11.637-05:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped

then I try to execute this curl command which is telling me to delete telemetry. but it fails


{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices

I even try deleting some old index but I can't do that anymore


# curl -u elastic:elastic -XDELETE houelkv8:9200/.kibana_task_manager_7.16.2_001
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with roles [superuser] on restricted indices [.kibana_task_manager_7.16.2_001], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}

and user:password is correct as I can see the cluster status

# curl -u elastic:elastic -XGET houelkv8:9200/_cluster/health?pretty
{
  "cluster_name" : "elk_v_8_0",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 48,
  "active_shards" : 48,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

How do I fix it?

elasticforme · May 17, 2022, 4:58pm

I don't even see .kibana_8.1.1_001 when I list _cat/indices

anyone has such issue?

ibra_013 · May 17, 2022, 8:24pm

Hi @elasticforme

Check this thread same issue as yours.

github.com/elastic/kibana

[Reporting] Implement allowing administrators to manage any report or reporting index

opened 02:56PM - 16 Mar 22 UTC

lucabelluccini

bug Feature:Reporting loe:multi-week Team:AppServicesSv impact:medium Team:Reporting Services

**Kibana version:** 8.0+ **Elasticsearch version:** aligned with Kibana **…Describe the bug:** On versions < 8.0, `.reporting*` indices could be deleted at least by the `superuser` or anyone who had `reporting_user` permissions. On 8.0+, due to the fact `.reporting*` indices are considered system indices, even a `superuser` do not have the permission to delete them (as `superuser` doesn't have the permission to delete restricted indices starting 8.0). **Steps to reproduce:** 1. Create report 2. Try to delete any `.reporting-*` index **Expected behavior:** A user should have the permission to delete reports and/or clean up old reports without having to use workarounds. The current workaround is to create a dedicated role and a user assigned to it which grants: ``` { "names" : [ ".reporting*" ], "privileges" : [ "monitor", "read", "view_index_metadata", "read_cross_cluster", "delete_index" ], "allow_restricted_indices" : true } ``` The problem occurs also when granting both `reporting_user` and `superuser` permissions. ``` { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "action [indices:admin/delete] is unauthorized for user [lucalucaluca] with roles [reporting_user,superuser] on indices [.reporting-2022-03-13], this action is granted by the index privileges [delete_index,manage,all]" } ], "type" : "security_exception", "reason" : "action [indices:admin/delete] is unauthorized for user [lucalucaluca] with roles [reporting_user,superuser] on indices [.reporting-2022-03-13], this action is granted by the index privileges [delete_index,manage,all]" }, "status" : 403 } ``` The same problem occurs when using the new Kibana permissions. Even granting the following to a given user, still blocks the removal of `.reporting*` indices. ``` { "reporting_my" : { "cluster" : [ "all" ], "indices" : [ { "names" : [ "kibana_sample_data_logs" ], "privileges" : [ "all" ], "field_security" : { "grant" : [ "*" ], "except" : [ ] }, "allow_restricted_indices" : false } ], "applications" : [ { "application" : "kibana-.kibana", "privileges" : [ "feature_discover.minimal_all", "feature_discover.url_create", "feature_discover.store_search_session", "feature_discover.generate_report", "feature_dashboard.minimal_all", "feature_dashboard.url_create", "feature_dashboard.store_search_session", "feature_dashboard.generate_report", "feature_dashboard.download_csv_report" ], "resources" : [ "*" ] } ], "run_as" : [ ], "metadata" : { }, "transient_metadata" : { "enabled" : true } } } ``` **Any additional context:** As `.reporting*` documents contains both the job/task definition and the "result" of the report, maybe we could decouple them into 2 separate indices? Also the `.reporting` index could become a "workspace" index (like `.async-search` index) where we keep only the jobs to be done and store the generated reports in a separate index. Or otherwise we should provide a dedicated API to manage reports `_reporting` at ES or Kibana level and allow to delete specific reports or whole groups of reports. ~I would consider it a bug as it is a regression on features of reporting~ > This as an effect of the breaking change made by ES security to no longer allow the superuser role to manage system indices by default. **PLEASE BE AWARE** that it is possible to auto-delete old Reporting indices modifying the ILM Policy `kibana-reporting`, adding a Delete phase. So, this problem affects only users who are interested in deleting a specific reporting index.

Luca_Belluccini · May 17, 2022, 8:57pm

Hello,

In order to help, I would like to ask if you can just start Elasticsearch and then check via curl, as elastic user:

GET _cat/indices/.kibana*?expand_wildcards=all&v

You should be able to see the index .kibana_8.1.1_001.
The index .kibana_8.2.0_001 might exist, but it will have less documents than 8.1.1.

The output of:

GET _cat/aliases/.kibana*?expand_wildcards=all&v

Should instead show .kibana pointing still to 8.1.1 indices.

In order to edit system indices in 8+, you need a special permission.
It is documented here.

Create a role as follows:

PUT _security/role/grant_kibana_indices
{
  "indices": [
    {
      "names": [
        ".kibana*"
      ],
      "privileges": [
        "all"
      ],
      "allow_restricted_indices": true
    }
  ]
}

Create a user with the role above and superuser built-in role:

POST /_security/user/temporarykibanasuperuser
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : [ "superuser", "grant_kibana_indices" ]
}

Run:

PUT .kibana_8.1.1_001/_settings
{
  "index": {
    "blocks.write": false
  }
}

Run (modify {elasticsearch} with the correct host), you will be prompted for a password:

curl -u temporarykibanasuperuser -X POST "{elasticsearch}/.kibana_8.1.1_001/_bulk?pretty" -H 'Content-Type: application/json' -d'{ "delete" : { "_id" : "telemetry:telemetry" } }'

Once done, start Kibana 8.2.0 and it should start and be able to finalize the Saved Objects migration.

To be honest, I've not seen other issues like this in the Kibana repository, so if you're able to reproduce this problem consistently, I would suggest filing an issue on the Kibana repository.

The procedure on how to solve Saved Objects issues is at this page, but it doesn't mention the need of creating a special permission.
I've opened [DOCS] Explain how to create a special user to solve Saved Object conflicts (edit Kibana index) · Issue #132370 · elastic/kibana · GitHub

elasticforme · May 17, 2022, 9:29pm

will this be fixed in next version?

I think I can produce this. if I redo 8.1.1 install and then upgrade again.

rudolf · May 18, 2022, 10:08am

@elasticforme The upgrade migrations fail because it seems like you disabled the telemetry plugin which results in the plugin not being loaded at all. As a result, Kibana doesn't know how to handle the data that belonged that plugin. The recommended way to disable telemetry is using telemetry.optIn: false Deprecate `telemetry.enabled: false` · Issue #129015 · elastic/kibana · GitHub The telemetry plugin will still be loaded, but it won't transmit any telemetry back to Elastic.

Now that you've removed the unknown saved object further upgrades won't run into this problem again so this won't affect you when e.g. upgrading to 8.2.0.

We also hope to make it easier to resolve upgrade failures like this by adding a new configuration flag Add migrations.ignoreUnknownObjects flag by gsoldevila · Pull Request #132167 · elastic/kibana · GitHub

elasticforme · May 18, 2022, 1:52pm

Thank you Rudolf. this makes complete sense.

system · June 15, 2022, 1:52pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.