hi
a have an 0.18.5 cluster of 4 servers, and problem with one of my shards
so there i active shard, but it also tryes to recover it on a slave node, and goes in loop
it tryes on one node, fails than on another, and constantly in slave i see next:
[16:43:27,936][WARN ][indices.cluster ] [Williams, Eric] [messages][5] failed to start shard
org.elasticsearch.indices.recovery.RecoveryFailedException: Index Shard [messages][5]: Recovery failed from [Bedlam][5K4U3Y-1S0e-yM7aqQvlmQ][inet[/10.174.45.231:9300]] into [Williams, Eric][g74hBddiS6O6CNXvJ1aWeg][inet[/10.176.247.143:9300]]
at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:263)
at org.elasticsearch.indices.recovery.RecoveryTarget.access$100(RecoveryTarget.java:73)
at org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:161)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.transport.RemoteTransportException: [Bedlam][inet[/10.174.45.231:9300]][index/shard/recovery/startRecovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [messages][5] Phase[2] Execution failed
at org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1105)
at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:491)
at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:93)
at org.elasticsearch.indices.recovery.RecoverySource.access$1200(RecoverySource.java:58)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:289)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:278)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:358)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.transport.RemoteTransportException: [Williams, Eric][inet[/10.176.247.143:9300]][index/shard/recovery/translogOps]
Caused by: org.elasticsearch.index.engine.IndexFailedEngineException: [messages][5] Index failed for [message#23334819635f7990b6e184d2]
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:482)
at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:572)
at org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:361)
at org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:347)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:358)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Mark invalid
at java.io.BufferedReader.reset(BufferedReader.java:485)
at org.apache.lucene.analysis.CharReader.reset(CharReader.java:69)
at org.elasticsearch.common.lucene.analysis.HTMLStripCharFilter.restoreState(HTMLStripCharFilter.java:177)
at org.elasticsearch.common.lucene.analysis.HTMLStripCharFilter.read(HTMLStripCharFilter.java:744)
at org.elasticsearch.common.lucene.analysis.HTMLStripCharFilter.read(HTMLStripCharFilter.java:759)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:822)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:1029)
at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:187)
at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:135)
at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:278)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:766)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2327)
at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:504)
at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:477)
... 7 more
[16:43:28,078][WARN ][cluster.action.shard ] [Williams, Eric] sending failed shard for [messages][5], node[g74hBddiS6O6CNXvJ1aWeg], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [messages][5]: Recovery failed from [Bedlam][5K4U3Y-1S0e-yM7aqQvlmQ][inet[/10.174.45.231:9300]] into [Williams, Eric][g74hBddiS6O6CNXvJ1aWeg][inet[/10.176.247.143:9300]]]; nested: RemoteTransportException[[Bedlam][inet[/10.174.45.231:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[messages][5] Phase[2] Execution failed]; nested: RemoteTransportException[[Williams, Eric][inet[/10.176.247.143:9300]][index/shard/recovery/translogOps]]; nested: IndexFailedEngineException[[messages][5] Index failed for [message#23334819635f7990b6e184d2]]; nested: IOException[Mark invalid]; ]]
on the shard's primary node there are next errors:
[15:33:12,647][WARN ][indices.cluster ] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message_content"},"cc":{"type":"string"},"company_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}]
org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message_content] not found for field [body]
at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86)
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162)
at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[15:53:36,501][WARN ][indices.cluster ] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message_content"},"cc":{"type":"string"},"company_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}]
org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message_content] not found for field [body]
at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86)
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162)
at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[16:05:11,021][WARN ][indices.cluster ] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message_content"},"cc":{"type":"string"},"company_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}]
org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message_content] not found for field [body]
at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86)
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162)
at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[16:13:27,408][WARN ][indices.cluster ] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message_content"},"cc":{"type":"string"},"company_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}]
org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message_content] not found for field [body]
at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86)
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162)
at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[16:14:51,666][WARN ][cluster.action.shard ] [Bedlam] received shard failed for [messages][5], node[dUFz1kbHSO6lmg1KxO5Obw], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [messages][5]: Recovery failed from [Bedlam][5K4U3Y-1S0e-yM7aqQvlmQ][inet[/10.174.45.231:9300]] into [Kamuu][dUFz1kbHSO6lmg1KxO5Obw][inet[/10.162.151.48:9300]]]; nested: RemoteTransportException[[Bedlam][inet[/10.174.45.231:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[messages][5] Phase[2] Execution failed]; nested: RemoteTransportException[[Kamuu][inet[/10.162.151.48:9300]][index/shard/recovery/translogOps]]; nested: IndexFailedEngineException[[messages][5] Index failed for [message#23334819635f7990b6e184d2]]; nested: IOException[Mark invalid]; ]]
[16:14:51,733][WARN ][indices.cluster ] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message_content"},"cc":{"type":"string"},"company_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}]
org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message_content] not found for field [body]
at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86)
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162)
at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[16:43:28,147][WARN ][cluster.action.shard ] [Bedlam] received shard failed for [messages][5], node[g74hBddiS6O6CNXvJ1aWeg], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [messages][5]: Recovery failed from [Bedlam][5K4U3Y-1S0e-yM7aqQvlmQ][inet[/10.174.45.231:9300]] into [Williams, Eric][g74hBddiS6O6CNXvJ1aWeg][inet[/10.176.247.143:9300]]]; nested: RemoteTransportException[[Bedlam][inet[/10.174.45.231:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[messages][5] Phase[2] Execution failed]; nested: RemoteTransportException[[Williams, Eric][inet[/10.176.247.143:9300]][index/shard/recovery/translogOps]]; nested: IndexFailedEngineException[[messages][5] Index failed for [message#23334819635f7990b6e184d2]]; nested: IOException[Mark invalid]; ]]
[16:43:28,181][WARN ][indices.cluster ] [Bedlam] [messages] failed to add mapping [message], source [{"message":{"_source":{"enabled":false},"properties":{"body":{"type":"string","analyzer":"message_content"},"cc":{"type":"string"},"company_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"date":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ssZ"},"id":{"type":"string"},"owner_id":{"type":"string","index":"not_analyzed","omit_term_freq_and_positions":true},"receiver":{"type":"string"},"sender":{"type":"string"},"subject":{"type":"string"}}}}]
org.elasticsearch.index.mapper.MapperParsingException: Analyzer [message_content] not found for field [body]
at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86)
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:84)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:260)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:221)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:162)
at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:263)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:393)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:346)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:271)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
and the sluster is in yellow state, alwasy trying to recover the shard on slaves:
{"cluster_name":"live","status":"yellow","timed_out":false,"number_of_nodes":4,"number_of_data_nodes":4,"active_primary_shards":17,"active_shards":33,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":0}
will the sluster go out of this state somehow, or it will be in the loop forever?