# River not reconnecting to RabbitMQ when accidently getting disconnected **URL:** \ **Category:** Elasticsearch\ **Created:** [June 8, 2012, 4:13pm UTC](https://discuss.elastic.co/t/river-not-reconnecting-to-rabbitmq-when-accidently-getting-disconnected/8022 "2012-06-08T16:13:16Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Max\_Kossatz](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@Max\_Kossatz](https://discuss.elastic.co/u/Max_Kossatz)\ **Post date:** [June 8, 2012, 4:13pm UTC](https://discuss.elastic.co/t/river-not-reconnecting-to-rabbitmq-when-accidently-getting-disconnected/8022/1 "2012-06-08T16:13:16Z")
Hi, we noticed that when the RabbitMQ-Server fails the river is giving this errormessage in the log: [2012-06-08 15:45:14,362][WARN][river.rabbitmq] [Lincoln, Lonnie Thompson] [rabbitmq][my\_river71] failed to ack [498547] com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285) at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872) at org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274) at java.lang.Thread.run(Thread.java:636) and then the river is not trying to reconnect (we waited at least 10 minutes). Is there a longer timeout or is this a bug? Thank you for your great product! Max ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 12, 2012, 9:25am UTC](https://discuss.elastic.co/t/river-not-reconnecting-to-rabbitmq-when-accidently-getting-disconnected/8022/2 "2012-06-12T09:25:30Z")
Seems like the connection to rabbitmq is closed, wondering how it happened... . Do you see any other messages? On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz [max.kossatz@gmail.com](mailto:max.kossatz@gmail.com) wrote: > Hi, > > we noticed that when the RabbitMQ-Server fails the river is giving this > errormessage in the log: > [2012-06-08 15:45:14,362][WARN][river.rabbitmq] [Lincoln, > Lonnie Thompson] [rabbitmq][my\_river71] failed to ack [498547] > com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; > reason: Attempt to use closed channel > at > com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) > at > com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) > at > com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285) > at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872) > at > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274) > at java.lang.Thread.run(Thread.java:636) > > and then the river is not trying to reconnect (we waited at least 10 > minutes). Is there a longer timeout or is this a bug? > > Thank you for your great product! > Max ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 12, 2012, 8:58pm UTC](https://discuss.elastic.co/t/river-not-reconnecting-to-rabbitmq-when-accidently-getting-disconnected/8022/3 "2012-06-12T20:58:53Z")
Hi Shay, It's look really like what I have already seen with the twitter river. After a failure, the river doesn't restart itself. And nothing in logs (only the failure with twitter). I don't know if you remember this thread (it was in April I think), you asked me to set the river log level to debug but we did not see anything strange. Could these issues be relative (Elasticsearch rivers internal) or not (relative to each river plugin) ? Thanks David 馃槈 Twitter : @dadoonet / @elasticsearchfr Le 12 juin 2012 脿 11:25, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) a 茅crit : > Seems like the connection to rabbitmq is closed, wondering how it happened... . Do you see any other messages? > > On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz [max.kossatz@gmail.com](mailto:max.kossatz@gmail.com) wrote: > Hi, > > we noticed that when the RabbitMQ-Server fails the river is giving this errormessage in the log: > [2012-06-08 15:45:14,362][WARN][river.rabbitmq] [Lincoln, Lonnie Thompson] [rabbitmq][my\_river71] failed to ack [498547] > com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel > at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) > at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) > at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285) > at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872) > at org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274) > at java.lang.Thread.run(Thread.java:636) > > and then the river is not trying to reconnect (we waited at least 10 minutes). Is there a longer timeout or is this a bug? > > Thank you for your great product! > Max ---
**Author:** ![Vladislav\_Pernin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vladislav_pernin/32/2452_2.png) [@Vladislav\_Pernin](https://discuss.elastic.co/u/Vladislav_Pernin)\ **Post date:** [March 6, 2013, 3:08pm UTC](https://discuss.elastic.co/t/river-not-reconnecting-to-rabbitmq-when-accidently-getting-disconnected/8022/4 "2013-03-06T15:08:15Z")
Hi, Has any of you found the solution ? A thread dump shows that the river is not stopped but waits in the LinkingBlockingQueue. Maybe using a nextDelivery with a timeout to force to client to reconnect, but I'm not sure that's the good solution ... Vladislav Le mardi 12 juin 2012 22:58:53 UTC+2, David Pilato a 茅crit : > Hi Shay, > > It's look really like what I have already seen with the twitter river. > After a failure, the river doesn't restart itself. And nothing in logs > (only the failure with twitter). > > I don't know if you remember this thread (it was in April I think), you > asked me to set the river log level to debug but we did not see anything > strange. > > Could these issues be relative (Elasticsearch rivers internal) or not > (relative to each river plugin) ? > > Thanks > David 馃槈 > Twitter : @dadoonet / @elasticsearchfr > > Le 12 juin 2012 脿 11:25, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) \\> a > 茅crit : > > Seems like the connection to rabbitmq is closed, wondering how it > happened... . Do you see any other messages? > > On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz \<[max.k...@gmail.com](mailto:max.k...@gmail.com)\ > > > wrote: > > > Hi, > > > > we noticed that when the RabbitMQ-Server fails the river is giving this > > errormessage in the log: > > [2012-06-08 15:45:14,362][WARN][river.rabbitmq] [Lincoln, > > Lonnie Thompson] [rabbitmq][my\_river71] failed to ack [498547] > > com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; > > reason: Attempt to use closed channel > > at > > com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) > > at > > com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) > > at > > com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285) > > at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872) > > at > > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274) > > at java.lang.Thread.run(Thread.java:636) > > > > and then the river is not trying to reconnect (we waited at least 10 > > minutes). Is there a longer timeout or is this a bug? > > > > Thank you for your great product! > > Max -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vladislav\_Pernin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vladislav_pernin/32/2452_2.png) [@Vladislav\_Pernin](https://discuss.elastic.co/u/Vladislav_Pernin)\ **Post date:** [March 7, 2013, 9:07am UTC](https://discuss.elastic.co/t/river-not-reconnecting-to-rabbitmq-when-accidently-getting-disconnected/8022/5 "2013-03-07T09:07:14Z")
Hi, After hours of debugging, I found that consumer.nextDelivery(bulkTimeout.millis()) throws a ShutdownSignalException. This RuntimeException is not catched by the catch block which only catch InterruptedException. So the thread exit silently. I filed a bug : [RabbitMQ River : does not reconnect after RabbitMQ restart 路 Issue #2741 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/2741) Vlad Le mercredi 6 mars 2013 16:08:15 UTC+1, Vladislav Pernin a 茅crit : > Hi, > > Has any of you found the solution ? > > A thread dump shows that the river is not stopped but waits in the > LinkingBlockingQueue. > > Maybe using a nextDelivery with a timeout to force to client to reconnect, > but I'm not sure that's the good solution ... > > Vladislav > > Le mardi 12 juin 2012 22:58:53 UTC+2, David Pilato a 茅crit : > > > Hi Shay, > > > > It's look really like what I have already seen with the twitter river. > > After a failure, the river doesn't restart itself. And nothing in logs > > (only the failure with twitter). > > > > I don't know if you remember this thread (it was in April I think), you > > asked me to set the river log level to debug but we did not see anything > > strange. > > > > Could these issues be relative (Elasticsearch rivers internal) or not > > (relative to each river plugin) ? > > > > Thanks > > David 馃槈 > > Twitter : @dadoonet / @elasticsearchfr > > > > Le 12 juin 2012 脿 11:25, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) a 茅crit : > > > > Seems like the connection to rabbitmq is closed, wondering how it > > happened... . Do you see any other messages? > > > > On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz [max.k...@gmail.com](mailto:max.k...@gmail.com) wrote: > > > > > Hi, > > > > > > we noticed that when the RabbitMQ-Server fails the river is giving this > > > errormessage in the log: > > > [2012-06-08 15:45:14,362][WARN][river.rabbitmq] [Lincoln, > > > Lonnie Thompson] [rabbitmq][my\_river71] failed to ack [498547] > > > com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; > > > reason: Attempt to use closed channel > > > at > > > com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) > > > at > > > com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) > > > at > > > com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285) > > > at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872) > > > at > > > org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274) > > > at java.lang.Thread.run(Thread.java:636) > > > > > > and then the river is not trying to reconnect (we waited at least 10 > > > minutes). Is there a longer timeout or is this a bug? > > > > > > Thank you for your great product! > > > Max -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 7, 2013, 9:12am UTC](https://discuss.elastic.co/t/river-not-reconnecting-to-rabbitmq-when-accidently-getting-disconnected/8022/6 "2013-03-07T09:12:24Z")
Very nice catch! -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 7 mars 2013 脿 10:07, Vladislav Pernin [vladislav.pernin@gmail.com](mailto:vladislav.pernin@gmail.com) a 茅crit : > Hi, > > After hours of debugging, I found that consumer.nextDelivery(bulkTimeout.millis()) throws a ShutdownSignalException. > This RuntimeException is not catched by the catch block which only catch InterruptedException. > So the thread exit silently. > > I filed a bug : [RabbitMQ River : does not reconnect after RabbitMQ restart 路 Issue #2741 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/2741) > > Vlad > > Le mercredi 6 mars 2013 16:08:15 UTC+1, Vladislav Pernin a 茅crit : > Hi, > > Has any of you found the solution ? > > A thread dump shows that the river is not stopped but waits in the LinkingBlockingQueue. > > Maybe using a nextDelivery with a timeout to force to client to reconnect, but I'm not sure that's the good solution ... > > Vladislav > > Le mardi 12 juin 2012 22:58:53 UTC+2, David Pilato a 茅crit : > Hi Shay, > > It's look really like what I have already seen with the twitter river. > After a failure, the river doesn't restart itself. And nothing in logs (only the failure with twitter). > > I don't know if you remember this thread (it was in April I think), you asked me to set the river log level to debug but we did not see anything strange. > > Could these issues be relative (Elasticsearch rivers internal) or not (relative to each river plugin) ? > > Thanks > David 馃槈 > Twitter : @dadoonet / @elasticsearchfr > > Le 12 juin 2012 脿 11:25, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) a 茅crit : > > > Seems like the connection to rabbitmq is closed, wondering how it happened... . Do you see any other messages? > > > > On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz [max.k...@gmail.com](mailto:max.k...@gmail.com) wrote: > > Hi, > > > > we noticed that when the RabbitMQ-Server fails the river is giving this errormessage in the log: > > [2012-06-08 15:45:14,362][WARN][river.rabbitmq] [Lincoln, Lonnie Thompson] [rabbitmq][my\_river71] failed to ack [498547] > > com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel > > at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) > > at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) > > at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285) > > at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872) > > at org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274) > > at java.lang.Thread.run(Thread.java:636) > > > > and then the river is not trying to reconnect (we waited at least 10 minutes). Is there a longer timeout or is this a bug? > > > > Thank you for your great product! > > Max > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:47am UTC](https://discuss.elastic.co/t/river-not-reconnecting-to-rabbitmq-when-accidently-getting-disconnected/8022/7 "2017-07-06T02:47:50Z")