Shard Failed: marking and sending shard failed due to [failed recovery]

Elastic Stack Elasticsearch
1

Hi,

I've installed a new windows ELK stack for testing, worked once but then got the below errors when trying to run again. I can see that there are issues with shards, but don't know how to fix this.


[2022-02-16T11:25:42,403][INFO ][o.e.g.GatewayService     ] [node-1] recovered [11] indices into cluster_state
[2022-02-16T11:25:46,331][WARN ][o.e.i.c.IndicesClusterStateService] [node-1] [.kibana_task_manager_7.17.0_001][0] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [.kibana_task_manager_7.17.0_001][0]: Recovery failed on {node-1}{tm3dtj09RLuP5nSKeEHfTg}{0Oy2oPY-QTOfi35ApdLHMQ}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=9205473280, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=4605345792}
        at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$21(IndexShard.java:3234) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$6(StoreRecovery.java:391) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:439) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:86) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2349) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.0.jar:7.17.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
        ... 11 more
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:159) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:414) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:460) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.seqno.ReplicationTracker.loadRetentionLeases(ReplicationTracker.java:468) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.loadRetentionLeases(IndexShard.java:2720) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2042) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2016) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:470) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.0.jar:7.17.0]
        ... 8 more
Caused by: java.io.IOException: failed to read C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st
        at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:409) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:460) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.seqno.ReplicationTracker.loadRetentionLeases(ReplicationTracker.java:468) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.loadRetentionLeases(IndexShard.java:2720) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2042) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2016) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:470) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.0.jar:7.17.0]
        ... 8 more
Caused by: org.elasticsearch.gateway.CorruptStateException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st")))
        at org.elasticsearch.gateway.MetadataStateFormat.read(MetadataStateFormat.java:309) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:405) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:460) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.seqno.ReplicationTracker.loadRetentionLeases(ReplicationTracker.java:468) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.loadRetentionLeases(IndexShard.java:2720) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2042) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2016) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:470) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.0.jar:7.17.0]
        ... 8 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="C:\Users\user\Downloads\elasticsearch-7.17.0-windows-x86_64\elasticsearch-7.17.0\data\nodes\0\indices\k0WKI0plQi--TKr67SznYA\0\_state\retention-leases-1451.st")))
        at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:523) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
        at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:547) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]
        at org.elasticsearch.gateway.MetadataStateFormat.read(MetadataStateFormat.java:287) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadGeneration(MetadataStateFormat.java:405) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestStateWithGeneration(MetadataStateFormat.java:435) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.gateway.MetadataStateFormat.loadLatestState(MetadataStateFormat.java:460) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.seqno.ReplicationTracker.loadRetentionLeases(ReplicationTracker.java:468) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.loadRetentionLeases(IndexShard.java:2720) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2042) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2016) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:470) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.0.jar:7.17.0]
        ... 8 more

Any help is really appreciated.

2

That would suggest that you might have some disk issues.

3

Thanks @warkolm for the feedback.

Regarding disk:

1- There are still plenty of free storage.
2- Windows is working normally and I'm not having any trouble with any other app.

What other issues might we think about?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.