# Failed Shard Recovery
**URL:** https://discuss.elastic.co/t/failed-shard-recovery/22073
**Category:** Elasticsearch
**Created:** [February 10, 2015, 8:55pm UTC](https://discuss.elastic.co/t/failed-shard-recovery/22073 "2015-02-10T20:55:59Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Mario\_Rodriguez](https://discuss.elastic.co/u/Mario_Rodriguez)
#### Post date: [February 10, 2015, 8:55pm UTC](https://discuss.elastic.co/t/failed-shard-recovery/22073/1 "2015-02-10T20:55:59Z")
I have 7 indices in our ES cluster, and one of the indices has an issue
that is preventing recovery.
Here is what I am seeing in the log:
[2015-02-10 00:00:02,483][WARN][indices.recovery] [ES\_Server1]
[prodcustomer][1] recovery from
[[ES\_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES\_Server2][inet[/5.5.5.5:9300]]]
failed
org.elasticsearch.transport.RemoteTransportException:
[ES\_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
[prodcustomer][1] Phase[1] Execution failed
at
org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
at
org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
at
org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
at
org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by:
org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
[prodcustomer][1] Failed to transfer [0] files with total size of [0b]
at
org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
at
org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
... 9 more
Caused by: java.io.EOFException: read past EOF:
MMapIndexInput(path="D:\ElasticSearchData\new\_es\_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
at
org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
at
org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
at
org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
... 10 more
[2015-02-10 00:00:02,483][WARN][indices.cluster] [ES\_Server1]
[prodcustomer][1] failed to start shard
org.elasticsearch.indices.recovery.RecoveryFailedException:
[prodcustomer][1]: Recovery failed from
[ES\_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES\_Server2][inet[/5.5.5.5:9300]] into
[ES\_Server1][R3ArnIZsSVSsd-VLEaU\_Ug][ES\_Server1][inet[ES\_Server1.verify.local/4.4.4.4:9300]]
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306)
at
org.elasticsearch.indices.recovery.RecoveryTarget.access$200(RecoveryTarget.java:65)
at
org.elasticsearch.indices.recovery.RecoveryTarget$3.run(RecoveryTarget.java:184)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.transport.RemoteTransportException:
[ES\_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
[prodcustomer][1] Phase[1] Execution failed
at
org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
at
org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
at
org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
at
org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by:
org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
[prodcustomer][1] Failed to transfer [0] files with total size of [0b]
at
org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
at
org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
... 9 more
Caused by: java.io.EOFException: read past EOF:
MMapIndexInput(path="D:\ElasticSearchData\new\_es\_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
at
org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
at
org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
at
org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
... 10 more
Is it as simple as deleting the checksum files?
Thank you for any insight anyone can provide.
Mario
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6c11de11-2d3b-4ed3-a173-bae4b50edf8c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c11de11-2d3b-4ed3-a173-bae4b50edf8c%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Mario\_Rodriguez](https://discuss.elastic.co/u/Mario_Rodriguez)
#### Post date: [February 10, 2015, 9:03pm UTC](https://discuss.elastic.co/t/failed-shard-recovery/22073/2 "2015-02-10T21:03:50Z")
We are using version 1.3.1
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d9b842a9-33ed-41f3-b8a3-a02f75e756d7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d9b842a9-33ed-41f3-b8a3-a02f75e756d7%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Mario\_Rodriguez](https://discuss.elastic.co/u/Mario_Rodriguez)
#### Post date: [February 11, 2015, 6:27pm UTC](https://discuss.elastic.co/t/failed-shard-recovery/22073/3 "2015-02-11T18:27:47Z")
Bumping up.
Does anyone have insight into this??
Mario
On Tuesday, February 10, 2015 at 3:02:12 PM UTC-6, Mario Rodriguez wrote:
> On Tuesday, February 10, 2015 at 2:55:59 PM UTC-6, Mario Rodriguez wrote:
>
> > I have 7 indices in our ES cluster, and one of the indices has an issue
> > that is preventing recovery.
> >
> > Here is what I am seeing in the log:
> >
> > [2015-02-10 00:00:02,483][WARN][indices.recovery] [ES\_Server1]
> > [prodcustomer][1] recovery from
> > [[ES\_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES\_Server2][inet[/5.5.5.5:9300]]]
> > failed
> > org.elasticsearch.transport.RemoteTransportException:
> > [ES\_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
> > Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
> > [prodcustomer][1] Phase[1] Execution failed
> > at
> > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
> > at
> > org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
> > at
> > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
> > Caused by:
> > org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
> > [prodcustomer][1] Failed to transfer [0] files with total size of [0b]
> > at
> > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
> > at
> > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
> > ... 9 more
> > Caused by: java.io.EOFException: read past EOF:
> > MMapIndexInput(path="D:\ElasticSearchData\new\_es\_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
> > at
> > org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
> > at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
> > at
> > org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
> > at
> > org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
> > at
> > org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
> > at
> > org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
> > at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
> > ... 10 more
> > [2015-02-10 00:00:02,483][WARN][indices.cluster] [ES\_Server1]
> > [prodcustomer][1] failed to start shard
> > org.elasticsearch.indices.recovery.RecoveryFailedException:
> > [prodcustomer][1]: Recovery failed from
> > [ES\_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES\_Server2][inet[/5.5.5.5:9300]]
> > into
> > [ES\_Server1][R3ArnIZsSVSsd-VLEaU\_Ug][ES\_Server1][inet[ES\_Server1.verify.local/
> > 4.4.4.4:9300]]
> > at
> > org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306)
> > at
> > org.elasticsearch.indices.recovery.RecoveryTarget.access$200(RecoveryTarget.java:65)
> > at
> > org.elasticsearch.indices.recovery.RecoveryTarget$3.run(RecoveryTarget.java:184)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
> > Caused by: org.elasticsearch.transport.RemoteTransportException:
> > [ES\_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
> > Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
> > [prodcustomer][1] Phase[1] Execution failed
> > at
> > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
> > at
> > org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
> > at
> > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
> > Caused by:
> > org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
> > [prodcustomer][1] Failed to transfer [0] files with total size of [0b]
> > at
> > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
> > at
> > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
> > ... 9 more
> > Caused by: java.io.EOFException: read past EOF:
> > MMapIndexInput(path="D:\ElasticSearchData\new\_es\_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
> > at
> > org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
> > at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
> > at
> > org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
> > at
> > org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
> > at
> > org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
> > at
> > org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
> > at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
> > at
> > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
> > ... 10 more
> >
> > Is it as simple as deleting the checksum files?
> > Thank you for any insight anyone can provide.
> >
> > \*EDIT We are on version 1.3.1
>
> > Mario
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7b527f9d-8d3a-4fce-a5b5-b030c7ab8994%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7b527f9d-8d3a-4fce-a5b5-b030c7ab8994%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@zivsegal](https://discuss.elastic.co/u/zivsegal)
#### Post date: [February 17, 2015, 10:15am UTC](https://discuss.elastic.co/t/failed-shard-recovery/22073/4 "2015-02-17T10:15:12Z")
+1
I have the same issue. working with 1.4.1
This is my stack trace:
[2015-02-17 09:36:08,438][WARN][indices.recovery] [data\_node]
[deragzicaqwarkgqdotwqecptkfibn-150216][1] recovery from
[[data\_node2][PC-hkqAnSQuPjzFjykEsSg][ip-X-X-X-X][inet[/x.x.x.x:9300]]{max\_local\_storage\_nodes=1,
zone=us-east-1d, master=false}] failed
org.elasticsearch.transport.RemoteTransportException:
[data\_node2][inet[/x.x.x.x:9300]][internal:index/shard/recovery/start\_recovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
[deragzicaqwarkgqdotwqecptkfibn-150216][1] Phase[2] Execution failed
at
org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1136)
at
org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:654)
at
org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:137)
at
org.elasticsearch.indices.recovery.RecoverySource.access$2600(RecoverySource.java:74)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:464)
at
org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:450)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException:
[data\_node][inet[/x.x.x.x:9300]][internal:index/shard/recovery/translog\_ops]
Caused by: org.elasticsearch.index.mapper.MapperParsingException: object
mapping for [json] tried to parse as object, but got EOF, has a concrete
value been provided to it?
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:498)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:541)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:490)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:392)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:775)
at
org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:433)
at
org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.
On Wednesday, February 11, 2015 at 8:27:47 PM UTC+2, Mario Rodriguez wrote:
> Bumping up.
> Does anyone have insight into this??
>
> Mario
>
> On Tuesday, February 10, 2015 at 3:02:12 PM UTC-6, Mario Rodriguez wrote:
>
> > On Tuesday, February 10, 2015 at 2:55:59 PM UTC-6, Mario Rodriguez wrote:
> >
> > > I have 7 indices in our ES cluster, and one of the indices has an issue
> > > that is preventing recovery.
> > >
> > > Here is what I am seeing in the log:
> > >
> > > [2015-02-10 00:00:02,483][WARN][indices.recovery] [ES\_Server1]
> > > [prodcustomer][1] recovery from
> > > [[ES\_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES\_Server2][inet[/5.5.5.5:9300]]]
> > > failed
> > > org.elasticsearch.transport.RemoteTransportException:
> > > [ES\_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
> > > Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
> > > [prodcustomer][1] Phase[1] Execution failed
> > > at
> > > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
> > > at
> > > org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
> > > at
> > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:744)
> > > Caused by:
> > > org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
> > > [prodcustomer][1] Failed to transfer [0] files with total size of [0b]
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
> > > at
> > > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
> > > ... 9 more
> > > Caused by: java.io.EOFException: read past EOF:
> > > MMapIndexInput(path="D:\ElasticSearchData\new\_es\_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
> > > at
> > > org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
> > > at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
> > > at
> > > org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
> > > at
> > > org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
> > > at
> > > org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
> > > at
> > > org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
> > > at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
> > > ... 10 more
> > > [2015-02-10 00:00:02,483][WARN][indices.cluster] [ES\_Server1]
> > > [prodcustomer][1] failed to start shard
> > > org.elasticsearch.indices.recovery.RecoveryFailedException:
> > > [prodcustomer][1]: Recovery failed from
> > > [ES\_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES\_Server2][inet[/5.5.5.5:9300]]
> > > into
> > > [ES\_Server1][R3ArnIZsSVSsd-VLEaU\_Ug][ES\_Server1][inet[ES\_Server1.verify.local/
> > > 4.4.4.4:9300]]
> > > at
> > > org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306)
> > > at
> > > org.elasticsearch.indices.recovery.RecoveryTarget.access$200(RecoveryTarget.java:65)
> > > at
> > > org.elasticsearch.indices.recovery.RecoveryTarget$3.run(RecoveryTarget.java:184)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:744)
> > > Caused by: org.elasticsearch.transport.RemoteTransportException:
> > > [ES\_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
> > > Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
> > > [prodcustomer][1] Phase[1] Execution failed
> > > at
> > > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
> > > at
> > > org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
> > > at
> > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:744)
> > > Caused by:
> > > org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
> > > [prodcustomer][1] Failed to transfer [0] files with total size of [0b]
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
> > > at
> > > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
> > > ... 9 more
> > > Caused by: java.io.EOFException: read past EOF:
> > > MMapIndexInput(path="D:\ElasticSearchData\new\_es\_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
> > > at
> > > org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
> > > at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
> > > at
> > > org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
> > > at
> > > org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
> > > at
> > > org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
> > > at
> > > org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
> > > at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
> > > at
> > > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
> > > ... 10 more
> > >
> > > Is it as simple as deleting the checksum files?
> > > Thank you for any insight anyone can provide.
> > >
> > > \*EDIT We are on version 1.3.1
> >
> > > Mario
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5aef93a0-8578-42a4-98b1-aec6580f529f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5aef93a0-8578-42a4-98b1-aec6580f529f%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@zivsegal](https://discuss.elastic.co/u/zivsegal)
#### Post date: [February 17, 2015, 11:37am UTC](https://discuss.elastic.co/t/failed-shard-recovery/22073/5 "2015-02-17T11:37:30Z")
BTW, following the suggestion
from [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/shard$20recovery$20fail/elasticsearch/138uWwW-vhM/WIKOhnI-DDYJ)
I was able to work around this issue by setting number\_of\_replicas to 0,
and than after verifying that all the replicas were gone, increasing it
back to 1. the new replicas came up well without any issues.
On Tuesday, February 17, 2015 at 12:15:12 PM UTC+2, Ziv Segal wrote:
> +1
>
> I have the same issue. working with 1.4.1
>
> This is my stack trace:
> [2015-02-17 09:36:08,438][WARN][indices.recovery] [data\_node]
> [deragzicaqwarkgqdotwqecptkfibn-150216][1] recovery from
> [[data\_node2][PC-hkqAnSQuPjzFjykEsSg][ip-X-X-X-X][inet[/x.x.x.x:9300]]{max\_local\_storage\_nodes=1,
> zone=us-east-1d, master=false}] failed
> org.elasticsearch.transport.RemoteTransportException:
> [data\_node2][inet[/x.x.x.x:9300]][internal:index/shard/recovery/start\_recovery]
> Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
> [deragzicaqwarkgqdotwqecptkfibn-150216][1] Phase[2] Execution failed
> at
> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1136)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:654)
> at
> org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:137)
> at
> org.elasticsearch.indices.recovery.RecoverySource.access$2600(RecoverySource.java:74)
> at
> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:464)
> at
> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:450)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.elasticsearch.transport.RemoteTransportException:
> [data\_node][inet[/x.x.x.x:9300]][internal:index/shard/recovery/translog\_ops]
> Caused by: org.elasticsearch.index.mapper.MapperParsingException: object
> mapping for [json] tried to parse as object, but got EOF, has a concrete
> value been provided to it?
> at
> org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:498)
> at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:541)
> at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:490)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:392)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:775)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:433)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.
>
> On Wednesday, February 11, 2015 at 8:27:47 PM UTC+2, Mario Rodriguez wrote:
>
> > Bumping up.
> > Does anyone have insight into this??
> >
> > Mario
> >
> > On Tuesday, February 10, 2015 at 3:02:12 PM UTC-6, Mario Rodriguez wrote:
> >
> > > On Tuesday, February 10, 2015 at 2:55:59 PM UTC-6, Mario Rodriguez wrote:
> > >
> > > > I have 7 indices in our ES cluster, and one of the indices has an issue
> > > > that is preventing recovery.
> > > >
> > > > Here is what I am seeing in the log:
> > > >
> > > > [2015-02-10 00:00:02,483][WARN][indices.recovery]
> > > > [ES\_Server1] [prodcustomer][1] recovery from
> > > > [[ES\_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES\_Server2][inet[/5.5.5.5:9300]]]
> > > > failed
> > > > org.elasticsearch.transport.RemoteTransportException:
> > > > [ES\_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
> > > > Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
> > > > [prodcustomer][1] Phase[1] Execution failed
> > > > at
> > > > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
> > > > at
> > > > org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
> > > > at
> > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:744)
> > > > Caused by:
> > > > org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
> > > > [prodcustomer][1] Failed to transfer [0] files with total size of [0b]
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
> > > > at
> > > > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
> > > > ... 9 more
> > > > Caused by: java.io.EOFException: read past EOF:
> > > > MMapIndexInput(path="D:\ElasticSearchData\new\_es\_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
> > > > at
> > > > org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
> > > > at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
> > > > at
> > > > org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
> > > > at
> > > > org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
> > > > at
> > > > org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
> > > > at
> > > > org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
> > > > at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
> > > > ... 10 more
> > > > [2015-02-10 00:00:02,483][WARN][indices.cluster]
> > > > [ES\_Server1] [prodcustomer][1] failed to start shard
> > > > org.elasticsearch.indices.recovery.RecoveryFailedException:
> > > > [prodcustomer][1]: Recovery failed from
> > > > [ES\_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES\_Server2][inet[/5.5.5.5:9300]]
> > > > into
> > > > [ES\_Server1][R3ArnIZsSVSsd-VLEaU\_Ug][ES\_Server1][inet[ES\_Server1.verify.local/
> > > > 4.4.4.4:9300]]
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoveryTarget.access$200(RecoveryTarget.java:65)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoveryTarget$3.run(RecoveryTarget.java:184)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:744)
> > > > Caused by: org.elasticsearch.transport.RemoteTransportException:
> > > > [ES\_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
> > > > Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
> > > > [prodcustomer][1] Phase[1] Execution failed
> > > > at
> > > > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
> > > > at
> > > > org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
> > > > at
> > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:744)
> > > > Caused by:
> > > > org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
> > > > [prodcustomer][1] Failed to transfer [0] files with total size of [0b]
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
> > > > at
> > > > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
> > > > ... 9 more
> > > > Caused by: java.io.EOFException: read past EOF:
> > > > MMapIndexInput(path="D:\ElasticSearchData\new\_es\_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
> > > > at
> > > > org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
> > > > at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
> > > > at
> > > > org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
> > > > at
> > > > org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
> > > > at
> > > > org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
> > > > at
> > > > org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
> > > > at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
> > > > at
> > > > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
> > > > ... 10 more
> > > >
> > > > Is it as simple as deleting the checksum files?
> > > > Thank you for any insight anyone can provide.
> > > >
> > > > \*EDIT We are on version 1.3.1
> > >
> > > > Mario
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bd771361-be04-4c98-b848-a4322e519f76%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bd771361-be04-4c98-b848-a4322e519f76%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:32am UTC](https://discuss.elastic.co/t/failed-shard-recovery/22073/6 "2017-07-06T00:32:12Z")