We have a four-node Elasticsearch cluster running ES 7.16.2 and Kibana 7.16.2. Since the last upgrade, our Kibana page is showing
Kibana server is not ready yet
The Kibana log shows this error repeated numerous times:
unavailable_shards_exception,reason:[.kibana_7.16.2_reindex_temp][0] Not enough active copies to meet shard count of [ALL] (have 1, needed 2)
We also have 1087 unassigned Elasticsearch shards.
This cluster is set up with a single replica for shards:
"number_of_shards": "1"
"number_of_replicas": "1"
"auto_expand_replicas": "0-1"
A search of this error showed that it's an orchestration issue where the shutdown metadata is not cleared (https://github.com/elastic/kibana/issues/127136), and the recommended workaround is to delete the shutdown metadata. But doing so results in another error telling me that none of the nodes is currently shutting down:
DELETE "_nodes/uFZB4oCySPqmp01KvTdPDw/shutdown"
{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"node [uFZB4oCySPqmp01KvTdPDw] is not currently shutting down"}],"type":"resource_not_found_exception","reason":"node [uFZB4oCySPqmp01KvTdPDw] is not currently shutting down"},"status":404}
This error is repeated for each of the four nodes.
At this point I'm kind of stuck for where to go from here.
Any ideas?