Kibana 8.8.0 won't start: Unable to complete saved object migrations for the [.kibana_analytics]

Elastic Stack Kibana
1

Just tried to upgrade to 8.8.0 and Kibana won't start.

Logs:

2024-06-07T12:45:06-07:00 [2024-06-07T19:45:06.172+00:00][INFO ][root] Kibana is shutting down
2024-06-07T12:45:06-07:00 [2024-06-07T19:45:06.172+00:00][FATAL][root] Reason: Unable to complete saved object migrations for the [.kibana_analytics] index. Error: [{"type":"unavailable_shards_exception","reason":"[.kibana_analytics_8.8.0_001][0] Not enough active copies to meet shard count of [ALL] (have 1, needed 2). Timeout: [1m], request: [BulkShardRequest [[.kibana_analytics_8.8.0_001][0]] containing [index {[.kibana_analytics_8.8.0_001][index-pattern:filebeat-*], source[n/a, actual length: [1.2mb], max length: 2kb]}]]"}]
2024-06-07T12:45:06-07:00 [Error: [{"type":"unavailable_shards_exception","reason":"[.kibana_analytics_8.8.0_001][0] Not enough active copies to meet shard count of [ALL] (have 1, needed 2). Timeout: [1m], request: [BulkShardRequest [[.kibana_analytics_8.8.0_001][0]] containing [index {[.kibana_analytics_8.8.0_001][index-pattern:filebeat-*], source[n/a, actual length: [1.2mb], max length: 2kb]}]]"}]
2024-06-07T12:45:06-07:00     at /usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/actions/bulk_overwrite_transformed_documents.js:74:13
2024-06-07T12:45:06-07:00     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-06-07T12:45:06-07:00     at stateActionMachine (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/state_action_machine.js:66:28)
2024-06-07T12:45:06-07:00     at migrationStateActionMachine (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/migrations_state_action_machine.js:47:24)
2024-06-07T12:45:06-07:00     at async Promise.all (index 2)
2024-06-07T12:45:06-07:00     at SavedObjectsService.start (/usr/share/kibana/node_modules/@kbn/core-saved-objects-server-internal/src/saved_objects_service.js:197:7)
2024-06-07T12:45:06-07:00     at Server.start (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/server.js:374:31)
2024-06-07T12:45:06-07:00     at Root.start (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/root/index.js:65:14)
2024-06-07T12:45:06-07:00     at bootstrap (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/bootstrap.js:118:5)
2024-06-07T12:45:06-07:00     at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:242:5)]
2024-06-07T12:45:06-07:00 [2024-06-07T19:45:06.177+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
2024-06-07T12:45:06-07:00 [2024-06-07T19:45:06.178+00:00][INFO ][plugins-system.preboot] All plugins stopped.
2024-06-07T12:45:06-07:00 [2024-06-07T19:45:06.178+00:00][INFO ][plugins-system.standard] Stopping all plugins.
2024-06-07T12:45:06-07:00 [2024-06-07T19:45:06.181+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped
2024-06-07T12:45:06-07:00 [2024-06-07T19:45:06.193+00:00][INFO ][plugins-system.standard] All plugins stopped.
2024-06-07T12:45:06-07:00 
2024-06-07T12:45:06-07:00  FATAL  Error: Unable to complete saved object migrations for the [.kibana_analytics] index. Error: [{"type":"unavailable_shards_exception","reason":"[.kibana_analytics_8.8.0_001][0] Not enough active copies to meet shard count of [ALL] (have 1, needed 2). Timeout: [1m], request: [BulkShardRequest [[.kibana_analytics_8.8.0_001][0]] containing [index {[.kibana_analytics_8.8.0_001][index-pattern:filebeat-*], source[n/a, actual length: [1.2mb], max length: 2kb]}]]"}]

Thinking I could maybe nuke the index and have it rebuilt:

$ curl -XDELETE https://elastic:--redacted--@elastic.--redacted--/.kibana_analytics_8.8.0_001
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with effective roles [superuser] on restricted indices [.kibana_analytics_8.8.0_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [elastic] with effective roles [superuser] on restricted indices [.kibana_analytics_8.8.0_001], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}

Seems odd that the 'superuser' account doesn't have permissions.

2

Oops. For some reason the kubernetes pods were deployed with a fraction of a CPU and somewhere in the mix something was timing out causing this error. Re-deploying it properly caused everything to spin right up.

3

Thanks for sharing your solution, @Aaron_C_de_Bruyn.

1 Like