# Rabbitmq river stops working **URL:** \ **Category:** Elasticsearch\ **Created:** [April 9, 2012, 3:21pm UTC](https://discuss.elastic.co/t/rabbitmq-river-stops-working/7273 "2012-04-09T15:21:17Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Max\_Kossatz](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@Max\_Kossatz](https://discuss.elastic.co/u/Max_Kossatz)\ **Post date:** [April 9, 2012, 3:21pm UTC](https://discuss.elastic.co/t/rabbitmq-river-stops-working/7273/1 "2012-04-09T15:21:17Z")
Hi, we are using elasticsearch for some time now and are very impressed how good it works. We are using ES in combination with rabbitmq and indexing around 1 million documents a day over this rabbitmq-river into ES. It works great! But from time to time (once a week) we get this "warning" in the logifle: [2012-04-09 12:19:15,724][WARN][river.rabbitmq] [Malekith the Accursed] [rabbitmq][my\_river8] failed to parse request for delivery tag [2965504], ack'ing... at org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:240) at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java:147) at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:92) at org.elasticsearch.action.bulk.BulkRequestBuilder.add(BulkRequestBuilder.java:81) at org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:240) at java.lang.Thread.run(Thread.java:636) And then the river stops to bulk insert documents into ES. We always have to restart the river by deleting the river and making it new, some times even under a different name, otherwise it will not connect to the rabbitmq server again. this are our river-settings: { "type":"rabbitmq", "rabbitmq":{ "host":"HOST\_IP", "port":5672, "user":"USERNAME", "pass":"PASSWORD", "vhost":"/", "queue":"SOME\_NAME", "exchange":"SOME\_NAME", "routing\_key":"SOME\_NAME", "exchange\_type":"SOME\_NAME", "exchange\_durable":true, "queue\_durable":true, "queue\_auto\_delete":false }, "index":{ "bulk\_size":100, "bulk\_timeout":"10s", "ordered":true } } Any ideas why? And is there any chance to see what bulkinsert made this error? Thank you & keep up the great work! Max Any ideas why? And why is the river not restarting again? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 11, 2012, 9:39am UTC](https://discuss.elastic.co/t/rabbitmq-river-stops-working/7273/2 "2012-04-11T09:39:25Z")
Strange... . I can see that it acks the failed message (which happens when the format of the bulk indexing message fails to parse). What I can see is that in this case, a delivery tag will be ack'ed twice, which I am not sure, once for the failure, and another later on, I can fix that, though I am not sure why this will cause rabbitmq to stop sending messages... On Mon, Apr 9, 2012 at 6:21 PM, Max Kossatz [max.kossatz@gmail.com](mailto:max.kossatz@gmail.com) wrote: > Hi, > > we are using elasticsearch for some time now and are very impressed how > good it works. > We are using ES in combination with rabbitmq and indexing around 1 million > documents a day over this rabbitmq-river into ES. > It works great! But from time to time (once a week) we get this "warning" > in the logifle: > > [2012-04-09 12:19:15,724][WARN][river.rabbitmq] [Malekith the > Accursed] [rabbitmq][my\_river8] failed to parse request for delivery tag > [2965504], ack'ing... > at > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:240) > at > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java:147) > at > org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:92) > at > org.elasticsearch.action.bulk.BulkRequestBuilder.add(BulkRequestBuilder.java:81) > at > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:240) > at java.lang.Thread.run(Thread.java:636) > > And then the river stops to bulk insert documents into ES. We always have > to restart the river by deleting the river and making it new, some times > even under a different name, otherwise it will not connect to the rabbitmq > server again. > > this are our river-settings: > { > "type":"rabbitmq", > "rabbitmq":{ > "host":"HOST\_IP", > "port":5672, > "user":"USERNAME", > "pass":"PASSWORD", > "vhost":"/", > "queue":"SOME\_NAME", > "exchange":"SOME\_NAME", > "routing\_key":"SOME\_NAME", > "exchange\_type":"SOME\_NAME", > "exchange\_durable":true, > "queue\_durable":true, > "queue\_auto\_delete":false > }, > "index":{ > "bulk\_size":100, > "bulk\_timeout":"10s", > "ordered":true > } > } > > Any ideas why? And is there any chance to see what bulkinsert made this > error? > Thank you & keep up the great work! > Max > > Any ideas why? And why is the river not restarting again? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 11, 2012, 9:42am UTC](https://discuss.elastic.co/t/rabbitmq-river-stops-working/7273/3 "2012-04-11T09:42:05Z")
Here is the issue: [https://github.com/elasticsearch/elasticsearch-river-rabbitmq/issues/9](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/issues/9). On Wed, Apr 11, 2012 at 12:39 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Strange... . I can see that it acks the failed message (which happens when > the format of the bulk indexing message fails to parse). What I can see is > that in this case, a delivery tag will be ack'ed twice, which I am not > sure, once for the failure, and another later on, I can fix that, though I > am not sure why this will cause rabbitmq to stop sending messages... > > On Mon, Apr 9, 2012 at 6:21 PM, Max Kossatz [max.kossatz@gmail.com](mailto:max.kossatz@gmail.com) wrote: > > > Hi, > > > > we are using elasticsearch for some time now and are very impressed how > > good it works. > > We are using ES in combination with rabbitmq and indexing around 1 > > million documents a day over this rabbitmq-river into ES. > > It works great! But from time to time (once a week) we get this "warning" > > in the logifle: > > > > [2012-04-09 12:19:15,724][WARN][river.rabbitmq] [Malekith the > > Accursed] [rabbitmq][my\_river8] failed to parse request for delivery tag > > [2965504], ack'ing... > > at > > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:240) > > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java:147) > > at > > org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:92) > > at > > org.elasticsearch.action.bulk.BulkRequestBuilder.add(BulkRequestBuilder.java:81) > > at > > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:240) > > at java.lang.Thread.run(Thread.java:636) > > > > And then the river stops to bulk insert documents into ES. We always have > > to restart the river by deleting the river and making it new, some times > > even under a different name, otherwise it will not connect to the rabbitmq > > server again. > > > > this are our river-settings: > > { > > "type":"rabbitmq", > > "rabbitmq":{ > > "host":"HOST\_IP", > > "port":5672, > > "user":"USERNAME", > > "pass":"PASSWORD", > > "vhost":"/", > > "queue":"SOME\_NAME", > > "exchange":"SOME\_NAME", > > "routing\_key":"SOME\_NAME", > > "exchange\_type":"SOME\_NAME", > > "exchange\_durable":true, > > "queue\_durable":true, > > "queue\_auto\_delete":false > > }, > > "index":{ > > "bulk\_size":100, > > "bulk\_timeout":"10s", > > "ordered":true > > } > > } > > > > Any ideas why? And is there any chance to see what bulkinsert made this > > error? > > Thank you & keep up the great work! > > Max > > > > Any ideas why? And why is the river not restarting again? ---
**Author:** ![Max\_Kossatz](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@Max\_Kossatz](https://discuss.elastic.co/u/Max_Kossatz)\ **Post date:** [April 12, 2012, 3:44pm UTC](https://discuss.elastic.co/t/rabbitmq-river-stops-working/7273/4 "2012-04-12T15:44:54Z")
Great, thank you very much for the fast help! Max On Wednesday, 11 April 2012 11:42:05 UTC+2, kimchy wrote: > Here is the issue: > [https://github.com/elasticsearch/elasticsearch-river-rabbitmq/issues/9](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/issues/9). > > On Wed, Apr 11, 2012 at 12:39 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Strange... . I can see that it acks the failed message (which happens > > when the format of the bulk indexing message fails to parse). What I can > > see is that in this case, a delivery tag will be ack'ed twice, which I am > > not sure, once for the failure, and another later on, I can fix that, > > though I am not sure why this will cause rabbitmq to stop sending > > messages... > > > > On Mon, Apr 9, 2012 at 6:21 PM, Max Kossatz [max.kossatz@gmail.com](mailto:max.kossatz@gmail.com)wrote: > > > > > Hi, > > > > > > we are using elasticsearch for some time now and are very impressed how > > > good it works. > > > We are using ES in combination with rabbitmq and indexing around 1 > > > million documents a day over this rabbitmq-river into ES. > > > It works great! But from time to time (once a week) we get this > > > "warning" in the logifle: > > > > > > [2012-04-09 12:19:15,724][WARN][river.rabbitmq] [Malekith > > > the Accursed] [rabbitmq][my\_river8] failed to parse request for delivery > > > tag [2965504], ack'ing... > > > at > > > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:240) > > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java:147) > > > at > > > org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:92) > > > at > > > org.elasticsearch.action.bulk.BulkRequestBuilder.add(BulkRequestBuilder.java:81) > > > at > > > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:240) > > > at java.lang.Thread.run(Thread.java:636) > > > > > > And then the river stops to bulk insert documents into ES. We always > > > have to restart the river by deleting the river and making it new, some > > > times even under a different name, otherwise it will not connect to the > > > rabbitmq server again. > > > > > > this are our river-settings: > > > { > > > "type":"rabbitmq", > > > "rabbitmq":{ > > > "host":"HOST\_IP", > > > "port":5672, > > > "user":"USERNAME", > > > "pass":"PASSWORD", > > > "vhost":"/", > > > "queue":"SOME\_NAME", > > > "exchange":"SOME\_NAME", > > > "routing\_key":"SOME\_NAME", > > > "exchange\_type":"SOME\_NAME", > > > "exchange\_durable":true, > > > "queue\_durable":true, > > > "queue\_auto\_delete":false > > > }, > > > "index":{ > > > "bulk\_size":100, > > > "bulk\_timeout":"10s", > > > "ordered":true > > > } > > > } > > > > > > Any ideas why? And is there any chance to see what bulkinsert made this > > > error? > > > Thank you & keep up the great work! > > > Max > > > > > > Any ideas why? And why is the river not restarting again? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:32am UTC](https://discuss.elastic.co/t/rabbitmq-river-stops-working/7273/5 "2017-07-06T03:32:48Z")