Kibana server is not ready yet, unavailable_shards_exception

We have a four-node Elasticsearch cluster running ES 7.16.2 and Kibana 7.16.2. Since the last upgrade, our Kibana page is showing

Kibana server is not ready yet

The Kibana log shows this error repeated numerous times:

unavailable_shards_exception,reason:[.kibana_7.16.2_reindex_temp][0] Not enough active copies to meet shard count of [ALL] (have 1, needed 2)

We also have 1087 unassigned Elasticsearch shards.

This cluster is set up with a single replica for shards:

"number_of_shards": "1"
"number_of_replicas": "1"
"auto_expand_replicas": "0-1"

A search of this error showed that it's an orchestration issue where the shutdown metadata is not cleared (https://github.com/elastic/kibana/issues/127136), and the recommended workaround is to delete the shutdown metadata. But doing so results in another error telling me that none of the nodes is currently shutting down:

DELETE "_nodes/uFZB4oCySPqmp01KvTdPDw/shutdown"

{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"node [uFZB4oCySPqmp01KvTdPDw] is not currently shutting down"}],"type":"resource_not_found_exception","reason":"node [uFZB4oCySPqmp01KvTdPDw] is not currently shutting down"},"status":404}

This error is repeated for each of the four nodes.

At this point I'm kind of stuck for where to go from here.

Any ideas?

Hi @Dave_G

Have you ever used the allocation explain API?
Cluster allocation explain API | Elasticsearch Guide [8.1] | Elastic

Ideally, your Elasticsearch cluster is healthy enough to connect to kibana.

best regards

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.