I'm having an issue with our Elastic Search / Kibana deployment. Kibana keeps failing to open with an error. Kibana 7.14.0_001 Windows
I was getting this fatal error:
log [12:18;05.466] [fatal][root] Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Error: [cluster.block_exception]: index [.kibana_task_manager_7.14.0_001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
I stopped getting it once I offboarded some indices. Now I am getting the following errors:
log [12:31:49.744] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
log [12:31:50.244] [info][savedobjects-service] Starting saved objects migrations
log [12:31:50.431] [info][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 56ms.
log [12:31:50.439] [info][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 62ms.
log [12:31:50.505] [error][savedobjects-service] [.kibana] Action failed with 'search_phase_execution_exception: '. Retrying attempt 1 in 2 seconds.
log [12:31:50.506] [info][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 74ms.
log [12:31:50.508] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 1 in 2 seconds.
log [12:31:50.509] [info][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 69ms.
log [12:31:52.583] [error][savedobjects-service] [.kibana] Action failed with 'search_phase_execution_exception: '. Retrying attempt 2 in 4 seconds.