# Shards failure - recovery possible?
**URL:** https://discuss.elastic.co/t/shards-failure-recovery-possible/231834
**Category:** Elasticsearch
**Created:** [May 9, 2020, 11:00am UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834 "2020-05-09T11:00:35Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Rysiu](https://discuss.elastic.co/u/Rysiu)
#### Post date: [May 9, 2020, 11:00am UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834/1 "2020-05-09T11:00:36Z")
Hi,
Unfortunately, there was a problem on my cluster when the power was switched off.
Cluster has RED status - a few shards do not want to get up.
I've tried using reroute:
`POST /_cluster/reroute?retry_failed=true`
but it's not working.
If I make an query:
```
GET /_cluster/allocation/explain
{
"index": "my-index-xxx",
"shard": 0,
"primary": true
}
```
The cluster will return:
```
{
"index" : "my-index-xxx",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2020-05-09T10:19:09.285Z",
"failed_allocation_attempts" : 5,
"details" : "failed shard on node [4oMUIeiASHeq8ZYNIx5hUg]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog from source [/var/lib/elasticsearch/nodes/0/indices/NiCEzOZjS9ib2oG2AC3QXg/0/translog/translog-134.tlog] is corrupted, translog truncated]; nested: EOFException[read past EOF. pos [1718122] length: [4] end: [1718122]]; ",
"last_allocation_status" : "no"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy",
"node_allocation_decisions" : [
{
"node_id" : "4oMUIeiASHeq8ZYNIx5hUg",
"node_name" : "data-006",
"transport_address" : "192.168.88.40:9300",
"node_attributes" : {
"machine_id" : "M001",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"in_sync" : true,
"allocation_id" : "f7brAMpUSgqLn1ERtP5teg"
},
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-05-09T10:19:09.285Z], failed_attempts[5], failed_nodes[[4oMUIeiASHeq8ZYNIx5hUg]], delayed=false, details[failed shard on node [4oMUIeiASHeq8ZYNIx5hUg]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog from source [/var/lib/elasticsearch/nodes/0/indices/NiCEzOZjS9ib2oG2AC3QXg/0/translog/translog-134.tlog] is corrupted, translog truncated]; nested: EOFException[read past EOF. pos [1718122] length: [4] end: [1718122]]; ], allocation_status[deciders_no]]]"
}
]
},
...
```
Any ideas on how to recover the shard?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [May 9, 2020, 1:09pm UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834/2 "2020-05-09T13:09:59Z")
This probably relates your other recent post:
> [@Possibilities of increasing data security in elasticsearch - power failures \[?\]](https://discuss.elastic.co/t/possibilities-of-increasing-data-security-in-elasticsearch-power-failures/231839):
>
> Hi, Relatively simple question about the configuration of elasticsearch with respect to specific data security. Are there any possibilities to configure an elasticsearch cluster so that in case of a power failure there are no additional data losses? What security models when not all nodes have redundant power supply? Is there anything we can do? Of course backup is backup. But I'm looking for a quick solution in elasticsearch.
Looks like your storage system does not, in fact, implement `fsync()` correctly.
What's the _whole_ response from the allocation explain API? It looks like all copies of this shard are broken. Does this index have replicas?
---
### Author:  [@Rysiu](https://discuss.elastic.co/u/Rysiu)
#### Post date: [May 9, 2020, 1:18pm UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834/3 "2020-05-09T13:18:02Z")
Unfortunately, this index had two replicas.
Each of the elasticsearch node has its own hard drive.
During a power outage, I lost power to the servers.
Despite the two replicas (a total (+primary) of three different/physical disks) the index have RED status.
Is something like that quite possible?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [May 9, 2020, 1:24pm UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834/4 "2020-05-09T13:24:49Z")
Repeating this request again:
> [@DavidTurner](#):
>
> What's the _whole_ response from the allocation explain API?
---
### Author:  [@Rysiu](https://discuss.elastic.co/u/Rysiu)
#### Post date: [May 9, 2020, 1:30pm UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834/5 "2020-05-09T13:30:31Z")
Full info:
```
{
"index" : "my-index-xxx",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2020-05-09T11:51:49.938Z",
"failed_allocation_attempts" : 5,
"details" : "failed shard on node [4oMUIeiASHeq8ZYNIx5hUg]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog from source [/var/lib/elasticsearch/nodes/0/indices/NiCEzOZjS9ib2oG2AC3QXg/0/translog/translog-134.tlog] is corrupted, translog truncated]; nested: EOFException[read past EOF. pos [1718122] length: [4] end: [1718122]]; ",
"last_allocation_status" : "no"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy",
"node_allocation_decisions" : [
{
"node_id" : "4oMUIeiASHeq8ZYNIx5hUg",
"node_name" : "data-006",
"transport_address" : "192.168.88.40:9300",
"node_attributes" : {
"machine_id" : "M001",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"in_sync" : true,
"allocation_id" : "f7brAMpUSgqLn1ERtP5teg"
},
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-05-09T11:51:49.938Z], failed_attempts[5], failed_nodes[[4oMUIeiASHeq8ZYNIx5hUg]], delayed=false, details[failed shard on node [4oMUIeiASHeq8ZYNIx5hUg]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog from source [/var/lib/elasticsearch/nodes/0/indices/NiCEzOZjS9ib2oG2AC3QXg/0/translog/translog-134.tlog] is corrupted, translog truncated]; nested: EOFException[read past EOF. pos [1718122] length: [4] end: [1718122]]; ], allocation_status[deciders_no]]]"
}
]
},
{
"node_id" : "5s8OBqvOSjOMrbwKgf0UVg",
"node_name" : "data-005",
"transport_address" : "192.168.88.39:9300",
"node_attributes" : {
"machine_id" : "M001",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "AisEn6iUT3q7PG08eVGIdg",
"node_name" : "data-000",
"transport_address" : "192.168.88.34:9300",
"node_attributes" : {
"machine_id" : "M000",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"in_sync" : false,
"allocation_id" : "tnMCOvcQTbOs40MRsQC-Hw"
}
},
{
"node_id" : "LAlFofPkRXCTb6AQzPfenQ",
"node_name" : "data-002",
"transport_address" : "192.168.88.36:9300",
"node_attributes" : {
"machine_id" : "M000",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "OAg_piYbQ5S2Lnyy582erg",
"node_name" : "data-001",
"transport_address" : "192.168.88.35:9300",
"node_attributes" : {
"machine_id" : "M000",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"in_sync" : false,
"allocation_id" : "L-sBscKGTjWe8EnFFNgxxg"
}
},
{
"node_id" : "gIZy3SQVSVCn6nMgMr51Bg",
"node_name" : "data-008",
"transport_address" : "192.168.88.42:9300",
"node_attributes" : {
"machine_id" : "M001",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "i45HkwMSTYGCyfwnJ1KFAg",
"node_name" : "data-004",
"transport_address" : "192.168.88.38:9300",
"node_attributes" : {
"machine_id" : "M000",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "jZyB7kNeS_WGI_Q6HgYj3A",
"node_name" : "data-007",
"transport_address" : "192.168.88.41:9300",
"node_attributes" : {
"machine_id" : "M001",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "loQtfthqRvqWNadLxAr5zA",
"node_name" : "data-003",
"transport_address" : "192.168.88.37:9300",
"node_attributes" : {
"machine_id" : "M000",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
}
]
}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [May 9, 2020, 1:52pm UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834/6 "2020-05-09T13:52:20Z")
Ok the other two copies of this shard were on nodes `data-000` and `data-001` but those copies are stale (i.e. do not contain all writes) possibly because those nodes failed first. Unfortunately that means there's no way to safely recover this data from within Elasticsearch.
I'd recommend deleting this index, restoring it from a recent snapshot, and then repeating any indexing that took place since the snapshot was taken. Alternatively you could perhaps recreate it from the source data assuming you've still got that.
---
### Author:  [@Rysiu](https://discuss.elastic.co/u/Rysiu)
#### Post date: [May 9, 2020, 3:10pm UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834/7 "2020-05-09T15:10:19Z")
That's what I'll do.
Thanks a lot!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 6, 2020, 3:10pm UTC](https://discuss.elastic.co/t/shards-failure-recovery-possible/231834/8 "2020-06-06T15:10:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.