# Shard Failed: marking and sending shard failed due to \[failed recovery\]

**URL:** <https://discuss.elastic.co/t/shard-failed-marking-and-sending-shard-failed-due-to-failed-recovery/297352>\
**Category:** Elasticsearch\
**Created:** [February 16, 2022, 10:22am UTC](https://discuss.elastic.co/t/shard-failed-marking-and-sending-shard-failed-due-to-failed-recovery/297352 "2022-02-16T10:22:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ethical20](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethical20/32/68123_2.png) [@ethical20](https://discuss.elastic.co/u/ethical20)\
**Post date:** [February 16, 2022, 10:22am UTC](https://discuss.elastic.co/t/shard-failed-marking-and-sending-shard-failed-due-to-failed-recovery/297352/1 "2022-02-16T10:22:42Z")

</div>

Hi,

I've installed a new windows ELK stack for testing, worked once but then got the below errors when trying to run again. I can see that there are issues with shards, but don't know how to fix this.

```auto

[2022-02-16T11:25:42,403][INFO][o.e.g.GatewayService] [node-1] recovered [11] indices into cluster_state
[2022-02-16T11:25:46,331][WARN][o.e.i.c.IndicesClusterStateService] [node-1] [.kibana_task_manager_7.17.0_001][0] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [.kibana_task_manager_7.17.0_001][0]: Recovery failed on {node-1}{tm3dtj09RLuP5nSKeEHfTg}{0Oy2oPY-QTOfi35ApdLHMQ}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=9205473280, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=4605345792}
        at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$21(IndexShard.java:3234) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$6(StoreRecovery.java:391) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:439) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:86) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2349) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.0.jar:7.17.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
        ... 11 more
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:159) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:414) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:460) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.seqno.ReplicationTracker.loadRetentionLeases(ReplicationTracker.java:468) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.loadRetentionLeases(IndexShard.java:2720) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2042) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2016) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:470) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.0.jar:7.17.0]
        ... 8 more
Caused by: java.io.IOException: failed to read C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st
        at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:409) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:460) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.seqno.ReplicationTracker.loadRetentionLeases(ReplicationTracker.java:468) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.loadRetentionLeases(IndexShard.java:2720) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2042) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2016) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:470) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.0.jar:7.17.0]
        ... 8 more
Caused by: org.elasticsearch.gateway.CorruptStateException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st")))
        at org.elasticsearch.gateway.MetadataStateFormat.read(MetadataStateFormat.java:309) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:405) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:460) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.seqno.ReplicationTracker.loadRetentionLeases(ReplicationTracker.java:468) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.loadRetentionLeases(IndexShard.java:2720) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2042) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2016) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:470) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.0.jar:7.17.0]
        ... 8 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st")))
        at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:523) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
        at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:547) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
        at org.elasticsearch.gateway.MetadataStateFormat.read(MetadataStateFormat.java:287) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:405) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:460) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.seqno.ReplicationTracker.loadRetentionLeases(ReplicationTracker.java:468) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.loadRetentionLeases(IndexShard.java:2720) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2042) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2016) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:470) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.0.jar:7.17.0]
        ... 8 more

```

Any help is really appreciated.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 16, 2022, 10:02pm UTC](https://discuss.elastic.co/t/shard-failed-marking-and-sending-shard-failed-due-to-failed-recovery/297352/2 "2022-02-16T22:02:39Z")

</div>

> [@ethical20](#):
>
> ```auto
> Caused by: org.elasticsearch.gateway.CorruptStateException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st")))
> 
> ```

That would suggest that you might have some disk issues.

---

<div class="post-metadata">

**Author:** ![ethical20](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethical20/32/68123_2.png) [@ethical20](https://discuss.elastic.co/u/ethical20)\
**Post date:** [February 17, 2022, 6:24am UTC](https://discuss.elastic.co/t/shard-failed-marking-and-sending-shard-failed-due-to-failed-recovery/297352/3 "2022-02-17T06:24:21Z")

</div>

Thanks @warkolm for the feedback.

Regarding disk:

1- There are still plenty of free storage.  
2- Windows is working normally and I'm not having any trouble with any other app.

What other issues might we think about?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 17, 2022, 6:25am UTC](https://discuss.elastic.co/t/shard-failed-marking-and-sending-shard-failed-due-to-failed-recovery/297352/4 "2022-03-17T06:25:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.