# Recovery failed for shard **URL:** \ **Category:** Elasticsearch\ **Created:** [December 27, 2011, 5:02pm UTC](https://discuss.elastic.co/t/recovery-failed-for-shard/6242 "2011-12-27T17:02:25Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![tetlika](https://avatars.discourse-cdn.com/v4/letter/t/c6cbf5/32.png) [@tetlika](https://discuss.elastic.co/u/tetlika)\ **Post date:** [December 27, 2011, 5:02pm UTC](https://discuss.elastic.co/t/recovery-failed-for-shard/6242/1 "2011-12-27T17:02:25Z")
hi a have an 0.18.5 cluster of 4 servers, and problem with one of my shards so there i active shard, but it also tryes to recover it on a slave node, and goes in loop it tryes on one node, fails than on another, and constantly in slave i see next: [16:43:27,936][WARN][indices.cluster] [Williams, Eric] [messages][5] failed to start shard org.elasticsearch.indices.recovery.RecoveryFailedException: Index Shard [messages][5]: Recovery failed from [Bedlam][5K4U3Y-1S0e-yM7aqQvlmQ][inet[/10.174.45.231:9300]] into [Williams, Eric][g74hBddiS6O6CNXvJ1aWeg][inet[/10.176.247.143:9300]] at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:263) at org.elasticsearch.indices.recovery.RecoveryTarget.access$100(RecoveryTarget.java:73) at org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:161) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.elasticsearch.transport.RemoteTransportException: [Bedlam][inet[/10.174.45.231:9300]][index/shard/recovery/startRecovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [messages][5] Phase[2] Execution failed at org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1105) at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:491) at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:93) at org.elasticsearch.indices.recovery.RecoverySource.access$1200(RecoverySource.java:58) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:289) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:278) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:358) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.elasticsearch.transport.RemoteTransportException: [Williams, Eric][inet[/10.176.247.143:9300]][index/shard/recovery/translogOps] Caused by: org.elasticsearch.index.engine.IndexFailedEngineException: [messages][5] Index failed for [message#23334819635f7990b6e184d2] at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:482) at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:572) at org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:361) at org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:347) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:358) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Mark invalid at java.io.BufferedReader.reset(BufferedReader.java:485) at org.apache.lucene.analysis.CharReader.reset(CharReader.java:69) at org.elasticsearch.common.lucene.analysis.HTMLStripCharFilter.restoreState(HTMLStripCharFilter.java:177) at org.elasticsearch.common.lucene.analysis.HTMLStripCharFilter.read(HTMLStripCharFilter.java:744) at org.elasticsearch.common.lucene.analysis.HTMLStripCharFilter.read(HTMLStripCharFilter.java:759) at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:822) at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:1029) at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:187) at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:135) at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:278) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:766) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2327) at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:504) at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:477) ... 7 more [16:43:28,078][WARN][cluster.action.shard] [Williams, Eric] sending failed shard for [messages][5], node[g74hBddiS6O6CNXvJ1aWeg], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [messages][5]: Recovery failed from [Bedlam][5K4U3Y-1S0e-yM7aqQvlmQ][inet[/10.174.45.231:9300]] into [Williams, Eric][g74hBddiS6O6CNXvJ1aWeg][inet[/10.176.247.143:9300]]]; nested: RemoteTransportException[[Bedlam][inet[/10.174.45.231:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[messages][5] Phase[2] Execution failed]; nested: RemoteTransportException[[Williams, Eric][inet[/10.176.247.143:9300]][index/shard/recovery/translogOps]]; nested: IndexFailedEngineException[[messages][5] Index failed for [message#23334819635f7990b6e184d2]]; nested: IOException[Mark invalid]; ]] on the shard's primary node there are next errors: [15:33:12,647][WARN][indices.cluster] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"\_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message\_content"},"cc":{"type":"string"},"company\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}] org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message\_content] not found for field [body] at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86) at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263) at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166) at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346) at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171) at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [15:53:36,501][WARN][indices.cluster] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"\_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message\_content"},"cc":{"type":"string"},"company\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}] org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message\_content] not found for field [body] at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86) at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263) at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166) at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346) at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171) at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [16:05:11,021][WARN][indices.cluster] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"\_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message\_content"},"cc":{"type":"string"},"company\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}] org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message\_content] not found for field [body] at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86) at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263) at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166) at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346) at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171) at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [16:13:27,408][WARN][indices.cluster] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"\_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message\_content"},"cc":{"type":"string"},"company\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}] org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message\_content] not found for field [body] at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86) at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263) at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166) at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346) at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171) at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [16:14:51,666][WARN][cluster.action.shard] [Bedlam] received shard failed for [messages][5], node[dUFz1kbHSO6lmg1KxO5Obw], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [messages][5]: Recovery failed from [Bedlam][5K4U3Y-1S0e-yM7aqQvlmQ][inet[/10.174.45.231:9300]] into [Kamuu][dUFz1kbHSO6lmg1KxO5Obw][inet[/10.162.151.48:9300]]]; nested: RemoteTransportException[[Bedlam][inet[/10.174.45.231:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[messages][5] Phase[2] Execution failed]; nested: RemoteTransportException[[Kamuu][inet[/10.162.151.48:9300]][index/shard/recovery/translogOps]]; nested: IndexFailedEngineException[[messages][5] Index failed for [message#23334819635f7990b6e184d2]]; nested: IOException[Mark invalid]; ]] [16:14:51,733][WARN][indices.cluster] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"\_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message\_content"},"cc":{"type":"string"},"company\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}] org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message\_content] not found for field [body] at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86) at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263) at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166) at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346) at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171) at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [16:43:28,147][WARN][cluster.action.shard] [Bedlam] received shard failed for [messages][5], node[g74hBddiS6O6CNXvJ1aWeg], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [messages][5]: Recovery failed from [Bedlam][5K4U3Y-1S0e-yM7aqQvlmQ][inet[/10.174.45.231:9300]] into [Williams, Eric][g74hBddiS6O6CNXvJ1aWeg][inet[/10.176.247.143:9300]]]; nested: RemoteTransportException[[Bedlam][inet[/10.174.45.231:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[messages][5] Phase[2] Execution failed]; nested: RemoteTransportException[[Williams, Eric][inet[/10.176.247.143:9300]][index/shard/recovery/translogOps]]; nested: IndexFailedEngineException[[messages][5] Index failed for [message#23334819635f7990b6e184d2]]; nested: IOException[Mark invalid]; ]] [16:43:28,181][WARN][indices.cluster] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"\_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message\_content"},"cc":{"type":"string"},"company\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner\_id":{"type":"string","index":"not\_analyzed","omit\_term\_freq\_and\_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}] org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message\_content] not found for field [body] at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86) at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260) at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221) at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162) at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263) at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166) at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393) at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346) at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171) at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) and the sluster is in yellow state, alwasy trying to recover the shard on slaves: {"cluster\_name":"live","status":"yellow","timed\_out":false,"number\_of\_nodes":4,"number\_of\_data\_nodes":4,"active\_primary\_shards":17,"active\_shards":33,"relocating\_shards":0,"initializing\_shards":1,"unassigned\_shards":0} will the sluster go out of this state somehow, or it will be in the loop forever? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:44am UTC](https://discuss.elastic.co/t/recovery-failed-for-shard/6242/2 "2017-07-06T03:44:18Z")