# Problems with RabbitMQ river & 0.90RC2 **URL:** \ **Category:** Elasticsearch\ **Created:** [April 27, 2013, 6:29pm UTC](https://discuss.elastic.co/t/problems-with-rabbitmq-river-0-90rc2/11716 "2013-04-27T18:29:42Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![max](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/max/32/2236_2.png) [@max](https://discuss.elastic.co/u/max)\ **Post date:** [April 27, 2013, 6:29pm UTC](https://discuss.elastic.co/t/problems-with-rabbitmq-river-0-90rc2/11716/1 "2013-04-27T18:29:42Z")
Hi, we have tried to upgrade our cluster to 0.90RC2 and everything worked fine (1.3 Billion Documents, 3.4TByte), the cluster was restarting fine and got into green status, flushing and refreshing did not returned any errors. But then we reactivated the RabbitMQ river (latest version 1.4.0) and suddenly shard where going offline and online all the time, leaving this messages in the errorlog: [2013-04-27 11:16:59,801][WARN][cluster.action.shard] [es-cluster-raid-5] sending failed shard for [egoarchive][6], node[jGrpxhH0Tj63U\_AuEw0hlQ], [P], s[STARTED], reason [engine failure, message [MergeException[org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [data]]; nested: ElasticSearchIllegalStateException[no index mapper found for field: [data]]; ]] [2013-04-27 11:16:59,803][WARN][index.merge.scheduler] [es-cluster-raid-5] [egoarchive][6] failed to merge org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [data] at org.elasticsearch.index.codec.PerFieldMappingPostingFormatCodec.getPostingsFormatForField(PerFieldMappingPostingFormatCodec.java:52) at org.apache.lucene.codecs.lucene42.Lucene42Codec$1.getPostingsFormatForField(Lucene42Codec.java:59) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:102) at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:71) at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401) at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478) [2013-04-27 11:17:02,521][WARN][indices.cluster] [es-cluster-raid-5] [egoarchive][6] master [[es-cluster-raid-3][rJ2\_qq3NRhaTmVFhm2ZMWg][inet[/10.55.65.214:9300]]] marked shard as started, but shard have not been created, mark shard as failed [2013-04-27 11:17:02,522][WARN][cluster.action.shard] [es-cluster-raid-5] sending failed shard for [egoarchive][6], node[jGrpxhH0Tj63U\_AuEw0hlQ], [P], s[STARTED], reason [master [es-cluster-raid-3][rJ2\_qq3NRhaTmVFhm2ZMWg][inet[/10.55.65.214:9300]] marked shard as started, but shard have not been created, mark shard as failed] [2013-04-27 11:17:04,246][WARN][indices.cluster] [es-cluster-raid-5] [egoarchive][6] master [[es-cluster-raid-3][rJ2\_qq3NRhaTmVFhm2ZMWg][inet[/10.55.65.214:9300]]] marked shard as started, but shard have not been created, mark shard as failed [2013-04-27 11:17:04,247][WARN][cluster.action.shard] [es-cluster-raid-5] sending failed shard for [egoarchive][6], node[jGrpxhH0Tj63U\_AuEw0hlQ], [P], s[STARTED], reason [master [es-cluster-raid-3][rJ2\_qq3NRhaTmVFhm2ZMWg][inet[/10.55.65.214:9300]] marked shard as started, but shard have not been created, mark shard as failed] [2013-04-27 11:17:45,654][WARN][index.merge.scheduler] [es-cluster-raid-5] [egoarchive][6] failed to merge org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [data] at org.elasticsearch.index.codec.PerFieldMappingPostingFormatCodec.getPostingsFormatForField(PerFieldMappingPostingFormatCodec.java:52) at org.apache.lucene.codecs.lucene42.Lucene42Codec$1.getPostingsFormatForField(Lucene42Codec.java:59) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:102) at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:71) at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401) at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478) So we where unable to use the river with 0.90RC2, the cluster was never stable running with the river connected, as soon as we disabled the river the cluster was stable again, any ideas why? Thank you, Max -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tommy\_Monk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tommy_monk/32/83712_2.png) [@Tommy\_Monk](https://discuss.elastic.co/u/Tommy_Monk)\ **Post date:** [May 13, 2013, 7:05am UTC](https://discuss.elastic.co/t/problems-with-rabbitmq-river-0-90rc2/11716/2 "2013-05-13T07:05:33Z")
Hi Max, We are also experiencing this same issue. For us the problem is occurring on ES 0.90 GA using the latest version of the Rabbit River. When rivering begins we see shards going offline and error logs such as; [2013-05-13 07:55:26,647][WARN][cluster.action.shard] [Allan, Liz] received shard failed for [global\_master][0], node[7oQb5ZkhRDGlNYtgAQQmrA], [P], s[STARTED], reason [engine failure, message [MergeException[org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [SourceProducts.MetaFields.OperatingSystem]]; nested: ElasticSearchIllegalStateException[no index mapper found for field: [SourceProducts.MetaFields.OperatingSystem]]; ]] [2013-05-13 07:55:26,648][WARN][indices.cluster] [Allan, Liz] [global\_master][0] master [[Allan, Liz][7oQb5ZkhRDGlNYtgAQQmrA][inet[/172.16.3.11:9300]]] marked shard as started, but shard have not been created, mark shard as failed Have you had any luck solving the issue ? Regards, -T On Saturday, 27 April 2013 19:29:42 UTC+1, Max Kossatz wrote: > Hi, > > we have tried to upgrade our cluster to 0.90RC2 and everything worked fine > (1.3 Billion Documents, 3.4TByte), the cluster was restarting fine and got > into green status, flushing and refreshing did not returned any errors. > > But then we reactivated the RabbitMQ river (latest version 1.4.0) and > suddenly shard where going offline and online all the time, leaving this > messages in the errorlog: > > [2013-04-27 11:16:59,801][WARN][cluster.action.shard] > [es-cluster-raid-5] sending failed shard for [egoarchive][6], > node[jGrpxhH0Tj63U\_AuEw0hlQ], [P], s[STARTED], reason [engine failure, > message > [MergeException[org.elasticsearch.ElasticSearchIllegalStateException: no > index mapper found for field: [data]]; nested: > ElasticSearchIllegalStateException[no index mapper found for field: > [data]]; ]] > [2013-04-27 11:16:59,803][WARN][index.merge.scheduler] > [es-cluster-raid-5] [egoarchive][6] failed to merge > org.elasticsearch.ElasticSearchIllegalStateException: no index mapper > found for field: [data] > at > org.elasticsearch.index.codec.PerFieldMappingPostingFormatCodec.getPostingsFormatForField(PerFieldMappingPostingFormatCodec.java:52) > at > org.apache.lucene.codecs.lucene42.Lucene42Codec$1.getPostingsFormatForField(Lucene42Codec.java:59) > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:102) > at > org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:71) > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383) > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401) > at > org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478) > [2013-04-27 11:17:02,521][WARN][indices.cluster] > [es-cluster-raid-5] [egoarchive][6] master > [[es-cluster-raid-3][rJ2\_qq3NRhaTmVFhm2ZMWg][inet[/10.55.65.214:9300]]] > marked shard as started, but shard have not been created, mark shard as > failed > [2013-04-27 11:17:02,522][WARN][cluster.action.shard] > [es-cluster-raid-5] sending failed shard for [egoarchive][6], > node[jGrpxhH0Tj63U\_AuEw0hlQ], [P], s[STARTED], reason [master > [es-cluster-raid-3][rJ2\_qq3NRhaTmVFhm2ZMWg][inet[/10.55.65.214:9300]] > marked shard as started, but shard have not been created, mark shard as > failed] > [2013-04-27 11:17:04,246][WARN][indices.cluster] > [es-cluster-raid-5] [egoarchive][6] master > [[es-cluster-raid-3][rJ2\_qq3NRhaTmVFhm2ZMWg][inet[/10.55.65.214:9300]]] > marked shard as started, but shard have not been created, mark shard as > failed > [2013-04-27 11:17:04,247][WARN][cluster.action.shard] > [es-cluster-raid-5] sending failed shard for [egoarchive][6], > node[jGrpxhH0Tj63U\_AuEw0hlQ], [P], s[STARTED], reason [master > [es-cluster-raid-3][rJ2\_qq3NRhaTmVFhm2ZMWg][inet[/10.55.65.214:9300]] > marked shard as started, but shard have not been created, mark shard as > failed] > [2013-04-27 11:17:45,654][WARN][index.merge.scheduler] > [es-cluster-raid-5] [egoarchive][6] failed to merge > org.elasticsearch.ElasticSearchIllegalStateException: no index mapper > found for field: [data] > at > org.elasticsearch.index.codec.PerFieldMappingPostingFormatCodec.getPostingsFormatForField(PerFieldMappingPostingFormatCodec.java:52) > at > org.apache.lucene.codecs.lucene42.Lucene42Codec$1.getPostingsFormatForField(Lucene42Codec.java:59) > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:102) > at > org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:71) > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383) > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401) > at > org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478) > > So we where unable to use the river with 0.90RC2, the cluster was never > stable running with the river connected, as soon as we disabled the river > the cluster was stable again, any ideas why? > > Thank you, > Max -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:36am UTC](https://discuss.elastic.co/t/problems-with-rabbitmq-river-0-90rc2/11716/3 "2017-07-06T02:36:58Z")