We moved our data to new version of elasticsearch. The new cluster have 3 master, 3 hot and 3 warm nodes. Everything was working fine till this morning and all of a cluster health went red. After looking at it further, I realized due to low storage on warm nodes, shard allocation failed. I increased…

Welcome to our community! :smiley: What version are you on? What do your master logs show? What does an ?explain show against one of the shards? What does _cat/recovery?v show?

Hi Mark, Thanks for your response. I don't see anything specific to the shard issue on master log for _cluster/allocation/explain { note: "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this…

/_cat/recovery?v index shard time type stage source_host source_node target_host target_node repository snapshot files files_recovered files_percent files_total bytes bytes_recovered bytes_percent bytes_total translog_ops translog_ops_recovered translog_o…

post some output of GET _cluster/settings GET _cluster/health mainly check routing allocation and how many shard is still unassigned and relocating

GET _cluster/health { cluster_name: "mainnet-es-cluster", status: "red", timed_out: false, number_of_nodes: 12, number_of_data_nodes: 9, active_primary_shards: 113, active_shards: 206, relocating_shards: 5, initializing_shards: 19, unassigned_shards: 63, delayed_unassigned_shards: 0, number_of_pend…

_cluster/settings persistent: { cluster: { routing: { allocation: { disk: { watermark: { low: "90%", high: "93%" } } } } } }, transient: { cluster: { routing: { allocation: { node_initial_primaries_recoveries: "6" } } } } }

Total of below 2 is always 82 initializing_shards: 19, unassigned_shards: 63,

This is the # of shards reinitializing but it doesn't work either initializing_shards: 19

Unable to recover my cluster

Elastic Stack Elasticsearch

elasticforme (Sachin) May 10, 2023, 4:07pm 10

what about
GET _cat/shards

this should show you many shard are reinitializing. if that is the case you have to wait

Topic		Replies	Views
Shards unassigned after node restarts - reason: NODE_LEFT Elasticsearch	16	37926	December 28, 2016
Could not reassign UNASSIGNED shards (elasticsearch 5.6 ) Elasticsearch	25	14708	January 25, 2018
Unassigned shards, v2 Elasticsearch	5	1356	July 6, 2017
Elastic `_cluster/health` showing unassigned shards Elasticsearch	2	596	July 20, 2023
Unassigned primary and replica shards Elasticsearch	6	2137	July 6, 2017

Unable to recover my cluster

Related topics