Kibana degraded after upgrade

[rudyfaile]

Hi,

I upgraded my ELK stack running on self-hosted kubernetes. My Elasticsearch cluster is green, but my kibana instance is spewing error logs like:

│ kibana [2023-11-28T15:24:58.959+00:00][ERROR][plugins.taskManager] Failed to poll for work: ResponseError: {"took":4,"timed_out":false," │
│ kibana [2023-11-28T15:25:01.961+00:00][ERROR][plugins.taskManager] Failed to poll for work: ResponseError: {"took":4,"timed_out":false," │
│ kibana [2023-11-28T15:25:04.960+00:00][ERROR][plugins.taskManager] Failed to poll for work: ResponseError: {"took":6,"timed_out":false," 
│ etc.

If I go to /status it shows kibana in a yellow state, sometimes flapping to unavailable and back to yellow in the logs.

I saw similar posts to my own, but did not see a resolution to any of them:

Kibana status is Yellow due to plugins degraded (after upgrade to version 8.8.2)

Hi, I just upgraded elasticseach/ kibana to 8.8.2. Cluster health is Green. But Kibana status is Yellow (when I navigate to http:///status ) What is see is there are lot of plugins in yellow status with the message degraded Can someone guide me how to resolve this issue ? [image] One of Kibana Log at kubernetes pod level - [2023-09-13T03:37:25.766+00:00][ERROR][plugins.alerting.metrics.alert.threshold] Executing Rule sales-and-service-cloud-monitoring:metrics.alert.threshold:4156c8df-16e…

Kibana Plugins Degraded

Hi there, I´ve done a few searchs for this kind of solution and I could not find anywhere. Can someone help me? [image] Jul 15 12:00:36 kbn kibana[3899]: [2022-07-15T12:00:36.438-03:00][ERROR][plugins.securitySolution] Bulk Indexing of signals failed: index: ".ds-logs-checkpoint.firewall-default-2022.07.14-000024" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illeg…

FWIW When I upgraded the vast majority of the issues I encountered were related to security stuff. The forced "secure by default" in ELK 8.x+ is very frustrating.

Thankful for any help I can get on this.

[rudyfaile]

The problem was this Upgrade to 8.7.0 hitting cluster_shard_limit_exceeded leads to write blocked index even after shard limit resolution · Issue #155136 · elastic/kibana · GitHub

even though we were nowhere near the limit. Somehow, blocks.write got set to true on an index, specifically: .kibana_task_manager_7.17.1_001. Removing the block fixed the problem.