# Marking and sending shard failed due to \[failed recovery\] **URL:** \ **Category:** Elasticsearch\ **Created:** [June 5, 2015, 3:06pm UTC](https://discuss.elastic.co/t/marking-and-sending-shard-failed-due-to-failed-recovery/2022 "2015-06-05T15:06:44Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![fernandomm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fernandomm/32/539_2.png) [@fernandomm](https://discuss.elastic.co/u/fernandomm)\ **Post date:** [June 5, 2015, 3:06pm UTC](https://discuss.elastic.co/t/marking-and-sending-shard-failed-due-to-failed-recovery/2022/1 "2015-06-05T15:06:44Z")
I'm testing ElasticSearch as a replacement for a big Mysql table ( over 1.3TB compressed ). While importing data I got the following error: ``` [2015-06-05 16:41:34,780][WARN][index.engine] [Venus Dee Milo] [logs][3] failed engine [already closed by tragic event] java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.codecs.lucene40.BitVector.clone(BitVector.java:78) at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.newLiveDocs(Lucene40LiveDocsFormat.java:85) at org.apache.lucene.index.ReadersAndUpdates.initWritableLiveDocs(ReadersAndUpdates.java:268) at org.apache.lucene.index.BufferedUpdatesStream.applyTermDeletes(BufferedUpdatesStream.java:441) at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:286) at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312) at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:420) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:171) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:568) at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:565) at org.elasticsearch.index.shard.IndexShard$EngineRefresher$1.run(IndexShard.java:1089) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ``` I saw that I needed to increase ES\_HEAP\_SIZE. I did and restarted ElasticSearch. But now I keep getting the following error: ``` [2015-06-05 16:56:28,752][WARN][cluster.action.shard] [Calvin Rankin] [logs][3] received shard failed for [logs][3], node[JbkJGd7XQV2-3ktQeGQXbA], [P], s[INITIALIZING], indexUUID [-RmOQ9o6RcOpSIKHxMp0fA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[logs][3] failed recovery]; nested: EngineCreationFailureException[[logs][3] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logs/3/index/segments_6f")]; ]] [2015-06-05 16:56:39,028][WARN][indices.cluster] [Calvin Rankin] [[logs][3]] marking and sending shard failed due to [failed recovery] org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logs][3] failed recovery at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [logs][3] failed to upgrade 3x segments at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:121) at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:24) at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1256) at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1251) at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:782) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112) ... 3 more Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logs/3/index/segments_6f") at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) at org.apache.lucene.store.DataInput.readInt(DataInput.java:98) at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183) at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:738) at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:749) at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1072) at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:119) ... 9 more ``` What can I do? Is everything that was imported to ElasticSearch lost? I'm running elasticsearch-1.5.1 in a Centos 6.6 box. ---
**Author:** ![DigiAngel](https://avatars.discourse-cdn.com/v4/letter/d/977dab/32.png) [@DigiAngel](https://discuss.elastic.co/u/DigiAngel)\ **Post date:** [June 8, 2015, 11:32pm UTC](https://discuss.elastic.co/t/marking-and-sending-shard-failed-due-to-failed-recovery/2022/2 "2015-06-08T23:32:31Z")
I have the exact same thing...did you get this resolved? ---
**Author:** ![fernandomm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fernandomm/32/539_2.png) [@fernandomm](https://discuss.elastic.co/u/fernandomm)\ **Post date:** [June 8, 2015, 11:56pm UTC](https://discuss.elastic.co/t/marking-and-sending-shard-failed-due-to-failed-recovery/2022/3 "2015-06-08T23:56:32Z")
Unfortunately no. I gave up using ElasticSearch for now since it seems to be very unreliable. I'm testing other alternatives that looks more promising than ES. ---
**Author:** ![DigiAngel](https://avatars.discourse-cdn.com/v4/letter/d/977dab/32.png) [@DigiAngel](https://discuss.elastic.co/u/DigiAngel)\ **Post date:** [June 9, 2015, 12:32am UTC](https://discuss.elastic.co/t/marking-and-sending-shard-failed-due-to-failed-recovery/2022/4 "2015-06-09T00:32:46Z")
Smart call. I have(had) a perfectly running 1.4.5 with no issues. After upgrading to 1.5, and doing a sudo service elasticsearch restart at noon that was it...I got the same as you. Luckily this was on a non-mission critical setup, so I downgraded back to 1.4.5, blew out the data and started over...now working like a champ. I would have to agree with you....ES 1.5 is not reliable. Hope you find a good solution. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:08am UTC](https://discuss.elastic.co/t/marking-and-sending-shard-failed-due-to-failed-recovery/2022/5 "2017-07-06T00:08:56Z")