Kibana not starting after upgrade to 8.17

Elastic Stack Kibana
1

Hello everybody
Since upgrading from 8.15.2 to 8.17.0 I've been unable to get Kibana to start, I've looked at other posts related to this but have been unable to resolve the issue.
Unfortunately I didn't use the upgrade assistant and I don't have a snapshot of the relevant indexes to restore.
I'm only seeing one error in the logs:

{"error":{"message":{"name":"ResponseError","message":"all shards failed: search_phase_execution_exception\n\tRoot causes:\n\t\tparse_exception: failed to parse date field [9223372036854776000] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [9223372036854776000] with format [strict_date_optional_time||epoch_millis]]"}},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-25T16:00:35.562+01:00","message":"Error happened during uninstall token generation.","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":13429,"uptime":26.430317483},"trace":{"id":"4f73f8396d0738f72e77a32bbbdfc0ad"},"transaction":{"id":"c9c9a08972be3fe8"}}

Any help would be appreciated
Thanks

2

did elasticsearch started ok?

what is the name is relevant index. log message do not have any name

3

Hello
Elasticsearch is working ok, status is green. I looked on the log entries immediately before and after but can't see an entry that specifies an index.
I have found more errors though:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.200+01:00","message":"Failed to attach alias to session index: Request timed out","log":{"level":"ERROR","logger":"plugins.security.session.index"},"process":{"pid":13429,"uptime":209031.068878442},"trace":{"id":"f153366705e104ed1a0efea23533be2b"},"transaction":{"id":"d792f82640464ecc"}}
{"tags":["b7971a11-4f94-11ec-9e8b-3b1e3382b93d","monitoring_alert_cluster_health"],"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.367+01:00","message":"error updating rule for monitoring_alert_cluster_health:b7971a11-4f94-11ec-9e8b-3b1e3382b93d Request timed out","log":{"level":"ERROR","logger":"plugins.alerting.monitoring_alert_cluster_health"},"process":{"pid":13429,"uptime":209031.234879991},"span":{"id":"44ed42adea93ba74"},"trace":{"id":"94e2e2ff19098abd163258315701b35d"}}
{"tags":["b7974120-4f94-11ec-9e8b-3b1e3382b93d","monitoring_alert_thread_pool_search_rejections"],"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.386+01:00","message":"error updating rule for monitoring_alert_thread_pool_search_rejections:b7974120-4f94-11ec-9e8b-3b1e3382b93d Request timed out","log":{"level":"ERROR","logger":"plugins.alerting.monitoring_alert_thread_pool_search_rejections"},"process":{"pid":13429,"uptime":209031.253943198},"span":{"id":"817f4a3d2593e829"},"trace":{"id":"94e2e2ff19098abd163258315701b35d"}}
{"tags":["b7967dd0-4f94-11ec-9e8b-3b1e3382b93d","monitoring_alert_cpu_usage"],"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.403+01:00","message":"error updating rule for monitoring_alert_cpu_usage:b7967dd0-4f94-11ec-9e8b-3b1e3382b93d Request timed out","log":{"level":"ERROR","logger":"plugins.alerting.monitoring_alert_cpu_usage"},"process":{"pid":13429,"uptime":209031.271506456},"span":{"id":"319e8f74b91324a4"},"trace":{"id":"94e2e2ff19098abd163258315701b35d"}}
{"tags":["b7f3b900-4f94-11ec-9e8b-3b1e3382b93d","monitoring_ccr_read_exceptions"],"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.418+01:00","message":"error updating rule for monitoring_ccr_read_exceptions:b7f3b900-4f94-11ec-9e8b-3b1e3382b93d Request timed out","log":{"level":"ERROR","logger":"plugins.alerting.monitoring_ccr_read_exceptions"},"process":{"pid":13429,"uptime":209031.286042826},"span":{"id":"af647768cd3575e1"},"trace":{"id":"94e2e2ff19098abd163258315701b35d"}}
{"tags":["1813cb5d-870c-537f-a45d-7bcb905f5aba","monitoring_alert_nodes_changed"],"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.435+01:00","message":"error updating rule for monitoring_alert_nodes_changed:1813cb5d-870c-537f-a45d-7bcb905f5aba Request timed out","log":{"level":"ERROR","logger":"plugins.alerting.monitoring_alert_nodes_changed"},"process":{"pid":13429,"uptime":209031.302831778},"trace":{"id":"08cac9d7dc147ebd445f3e4b7fa30d54"},"transaction":{"id":"85967f47573a5147"}}
{"tags":["b6733322-2bf2-5da9-bd19-bc4549f9e870","monitoring_alert_missing_monitoring_data"],"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.452+01:00","message":"error updating rule for monitoring_alert_missing_monitoring_data:b6733322-2bf2-5da9-bd19-bc4549f9e870 Request timed out","log":{"level":"ERROR","logger":"plugins.alerting.monitoring_alert_missing_monitoring_data"},"process":{"pid":13429,"uptime":209031.320681225},"trace":{"id":"f157c05c463c70611b7a6bf49edd5cb7"},"transaction":{"id":"063c0c428ec0de48"}}
{"tags":["b796f301-4f94-11ec-9e8b-3b1e3382b93d","monitoring_alert_missing_monitoring_data"],"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.638+01:00","message":"error updating rule for monitoring_alert_missing_monitoring_data:b796f301-4f94-11ec-9e8b-3b1e3382b93d Request timed out","log":{"level":"ERROR","logger":"plugins.alerting.monitoring_alert_missing_monitoring_data"},"process":{"pid":13429,"uptime":209031.506419701},"span":{"id":"18f3386fc80c6d35"},"trace":{"id":"94e2e2ff19098abd163258315701b35d"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.727+01:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":7,\"total_all_statuses\":7,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"inactive\":0,\"unenrolled\":0,\"num_host_urls\":0},\"license_issued_to\":\"networks01\"}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":13429,"uptime":209031.594951338},"trace":{"id":"b554cf9cbdc04180ae809f44072fe8ed"},"transaction":{"id":"8b6e770819d35a55"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-04-28T02:04:00.814+01:00","message":"error writing bulk events: \"Request timed out\"; docs: [{\"create\":{}},{\"@timestamp\":\"2025-04-28T01:03:30.260Z\",\"event\":{\"provider\":\"alerting\",\"action\":\"execute-start\",\"kind\":\"alert\",\"category\":[\"monitoring\"],\"start\":\"2025-04-28T01:03:30.260Z\"},\"kibana\":{\"alert\":{\"rule\":{\"rule_type_id\":\"monitoring_ccr_read_exceptions\",\"execution\":{\"uuid\":\"8182fab3-3438-407c-bdbb-7b0b81f2ebc4\"}}},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"b7f3b900-4f94-11ec-9e8b-3b1e3382b93d\",\"type_id\":\"monitoring_ccr_read_exceptions\"}],\"space_ids\":[\"default\"],\"task\":{\"scheduled\":\"2025-04-28T01:02:51.175Z\",\"schedule_delay\":39085000000},\"server_uuid\":\"66ceb11d-2f91-4357-88fb-b70f3fa18fe9\",\"version\":\"8.17.0\"},\"rule\":{\"id\":\"b7f3b900-4f94-11ec-9e8b-3b1e3382b93d\",\"license\":\"basic\",\"category\":\"monitoring_ccr_read_exceptions\",\"ruleset\":\"monitoring\"},\"message\":\"rule execution start: \\\"b7f3b900-4f94-11ec-9e8b-3b1e3382b93d\\\"\",\"ecs\":{\"version\":\"1.8.0\"}},{\"create\":{}},{\"@timestamp\":\"2025-04-28T01:03:30.261Z\",\"event\":{\"provider\":\"alerting\",\"action\":\"execute-start\",\"kind\":\"alert\",\"category\":[\"monitoring\"],\"start\":\"2025-04-28T01:03:30.261Z\"},\"kibana\":{\"alert\":{\"rule\":{\"rule_type_id\":\"monitoring_alert_cpu_usage\",\"execution\":{\"uuid\":\"cc729aa1-783e-4740-95ed-24b8010a94c0\"}}},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"b7967dd0-4f94-11ec-9e8b-3b1e3382b93d\",\"type_id\":\"monitoring_alert_cpu_usage\"}],\"space_ids\":[\"default\"],\"task\":{\"scheduled\":\"2025-04-28T01:02:51.175Z\",\"schedule_delay\":39086000000},\"server_uuid\":\"66ceb11d-2f91-4357-88fb-b70f3fa18fe9\",\"version\":\"8.17.0\"},\"rule\":{\"id\":\"b7967dd0-4f94-11ec-9e8b-3b1e3382b93d\",\"license\":\"basic\",\"category\":\"monitoring_alert_cpu_usage\",\"ruleset\":\"monitoring\"},\"message\":\"rule execution start: \\\"b7967dd0-4f94-11ec-9e8b-3b1e3382b93d\\\"\",\"ecs\":{\"version\":\"1.8.0\"}},{\"create\":{}},{\"@timestamp\":\"2025-04-28T01:03:30.262Z\",\"event\":{\"provider\":\"alerting\",\"action\":\"execute-start\",\"kind\":\"alert\",\"category\":[\"monitoring\"],\"start\":\"2025-04-28T01:03:30.262Z\"},\"kibana\":{\"alert\":{\"rule\":{\"rule_type_id\":\"monitoring_alert_thread_pool_search_rejections\",\"execution\":{\"uuid\":\"9035abc3-80dc-4136-a58c-064deb1bf52c\"}}},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"b7974120-4f94-11ec-9e8b-3b1e3382b93d\",\"type_id\":\"monitoring_alert_thread_pool_search_rejections\"}],\"space_ids\":[\"default\"],\"task\":{\"scheduled\":\"2025-04-28T01:02:51.175Z\",\"schedule_delay\":39087000000},\"server_uuid\":\"66ceb11d-2f91-4357-88fb-b70f3fa18fe9\",\"version\":\"8.17.0\"},\"rule\":{\"id\":\"b7974120-4f94-11ec-9e8b-3b1e3382b93d\",\"license\":\"basic\",\"category\":\"monitoring_alert_thread_pool_search_rejections\",\"ruleset\":\"monitoring\"},\"message\":\"rule execution start: \\\"b7974120-4f94-11ec-9e8b-3b1e3382b93d\\\"\",\"ecs\":{\"version\":\"1.8.0\"}},{\"create\":{}},{\"@timestamp\":\"2025-04-28T01:03:30.262Z\",\"event\":{\"provider\":\"alerting\",\"action\":\"execute-start\",\"kind\":\"alert\",\"category\":[\"monitoring\"],\"start\":\"2025-04-28T01:03:30.262Z\"},\"kibana\":{\"alert\":{\"rule\":{\"rule_type_id\":\"monitoring_alert_nodes_changed\",\"execution\":{\"uuid\":\"25d37aa9-04ad-4605-9ca0-01f94bf1eb64\"}}},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"1813cb5d-870c-537f-a45d-7bcb905f5aba\",\"type_id\":\"monitoring_alert_nodes_changed\",\"namespace\":\"sophos-xg\"}],\"space_ids\":[\"sophos-xg\"],\"task\":{\"scheduled\":\"2025-04-28T01:02:51.175Z\",\"schedule_delay\":39087000000},\"server_uuid\":\"66ceb11d-2f91-4357-88fb-b70f3fa18fe9\",\"version\":\"8.17.0\"},\"rule\":{\"id\":\"1813cb5d-870c-537f-a45d-7bcb905f5aba\",\"license\":\"basic\",\"category\":\"monitoring_alert_nodes_changed\",\"ruleset\":\"monitoring\"},\"message\":\"rule execution start: \\\"1813cb5d-870c-537f-a45d-7bcb905f5aba\\\"\",\"ecs\":{\"version\":\"1.8.0\"}},{\"create\":{}},{\"@timestamp\":\"2025-04-28T01:03:30.263Z\",\"event\":{\"provider\":\"alerting\",\"action\":\"execute-start\",\"kind\":\"alert\",\"category\":[\"monitoring\"],\"start\":\"2025-04-28T01:03:30.263Z\"},\"kibana\":{\"alert\":{\"rule\":{\"rule_type_id\":\"monitoring_alert_cluster_health\",\"execution\":{\"uuid\":\"f271ddd7-17d5-4c36-a3b4-37dcf0c37c72\"}}},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"b7971a11-4f94-11ec-9e8b-3b1e3382b93d\",\"type_id\":\"monitoring_alert_cluster_health\"}],\"space_ids\":[\"default\"],\"task\":{\"scheduled\":\"2025-04-28T01:02:51.175Z\",\"schedule_delay\":39088000000},\"server_uuid\":\"66ceb11d-2f91-4357-88fb-b70f3fa18fe9\",\"version\":\"8.17.0\"},\"rule\":{\"id\":\"b7971a11-4f94-11ec-9e8b-3b1e3382b93d\",\"license\":\"basic\",\"category\":\"monitoring_alert_cluster_health\",\"ruleset\":\"monitoring\"},\"message\":\"rule execution start: \\\"b7971a11-4f94-11ec-9e8b-3b1e3382b93d\\\"\",\"ecs\":{\"version\":\"1.8.0\"}},{\"create\":{}},{\"@timestamp\":\"2025-04-28T01:03:30.265Z\",\"event\":{\"provider\":\"alerting\",\"action\":\"execute-start\",\"kind\":\"alert\",\"category\":[\"monitoring\"],\"start\":\"2025-04-28T01:03:30.265Z\"},\"kibana\":{\"alert\":{\"rule\":{\"rule_type_id\":\"monitoring_alert_missing_monitoring_data\",\"execution\":{\"uuid\":\"169fd55d-cff5-49a2-bbc5-d09b591b9d76\"}}},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"b6733322-2bf2-5da9-bd19-bc4549f9e870\",\"type_id\":\"monitoring_alert_missing_monitoring_data\",\"namespace\":\"sophos-xg\"}],\"space_ids\":[\"sophos-xg\"],\"task\":{\"scheduled\":\"2025-04-28T01:02:51.670Z\",\"schedule_delay\":38595000000},\"server_uuid\":\"66ceb11d-2f91-4357-88fb-b70f3fa18fe9\",\"version\":\"8.17.0\"},\"rule\":{\"id\":\"b6733322-2bf2-5da9-bd19-bc4549f9e870\",\"license\":\"basic\",\"category\":\"monitoring_alert_missing_monitoring_data\",\"ruleset\":\"monitoring\"},\"message\":\"rule execution start: \\\"b6733322-2bf2-5da9-bd19-bc4549f9e870\\\"\",\"ecs\":{\"version\":\"1.8.0\"}},{\"create\":{}},{\"@timestamp\":\"2025-04-28T01:03:30.555Z\",\"event\":{\"provider\":\"alerting\",\"action\":\"execute-start\",\"kind\":\"alert\",\"category\":[\"monitoring\"],\"start\":\"2025-04-28T01:03:30.555Z\"},\"kibana\":{\"alert\":{\"rule\":{\"rule_type_id\":\"monitoring_alert_missing_monitoring_data\",\"execution\":{\"uuid\":\"3db2ba1a-b4e9-4b10-b21c-431804ac666e\"}}},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"b796f301-4f94-11ec-9e8b-3b1e3382b93d\",\"type_id\":\"monitoring_alert_missing_monitoring_data\"}],\"space_ids\":[\"default\"],\"task\":{\"scheduled\":\"2025-04-28T01:02:52.127Z\",\"schedule_delay\":38428000000},\"server_uuid\":\"66ceb11d-2f91-4357-88fb-b70f3fa18fe9\",\"version\":\"8.17.0\"},\"rule\":{\"id\":\"b796f301-4f94-11ec-9e8b-3b1e3382b93d\",\"license\":\"basic\",\"category\":\"monitoring_alert_missing_monitoring_data\",\"ruleset\":\"monitoring\"},\"message\":\"rule execution start: \\\"b796f301-4f94-11ec-9e8b-3b1e3382b93d\\\"\",\"ecs\":{\"version\":\"1.8.0\"}}]","log":{"level":"ERROR","logger":"plugins.eventLog"},"process":{"pid":13429,"uptime":209031.682637279},"trace":{"id":"f153366705e104ed1a0efea23533be2b"},"transaction":{"id":"d792f82640464ecc"}}

Thanks

4

something to do with updating rule? did you had alert rule enable in previous version?

5

As Sachin said, check ES. Please check the elastic cluster log, there should be something.
You have Request timed out