Shards are in ALLOCATION_FAILED or CLUSTER_RECOVERED

[Sathish22 (Sathish Thumma)]

Hi All,

we have a 5 master and 41 node cluster, due to some storage hardware issue, Elasticsearch cluster was affected and after resolving hardware issue, some of the shards are showing as ALLOCATION_FAILED or CLUSTER_RECOVERED in the cluster settings it is showing as

CLUSTER_RECOVERED: "cannot allocate because all found copies of the shard are either stale or corrupt"
ALLOCATION_FAILED: "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy"

can anyone please help us what action we need to take to gets this shards to assigned state

[Christian_Dahlqvist]

It sounds like you will need to restore the affected indices from a recent snapshot in order not to lise any data. How many primary shards are unassigned if you check the _cat/shards API?

[Sathish22 (Sathish Thumma)]

Hi Christian,

we don't have any snapshot configured in our system and there is no replicas configured and there are around 20 unassigned shards with either CLUSTER_RECOVERED/ALLOCATION_FAILED as unassigned.reason

is there any possibility to get them assigned/recovered ?

[Christian_Dahlqvist]

You can force allocation of the unassigned shards as oulined in the docs but that is likely to lead to data loss. If you run without replica shards and snapshots you are running the risk of data loss in cases like this.