River not reconnecting to RabbitMQ when accidently getting disconnected

Max_Kossatz · June 8, 2012, 4:13pm

Hi,

we noticed that when the RabbitMQ-Server fails the river is giving this
errormessage in the log:
[2012-06-08 15:45:14,362][WARN ][river.rabbitmq ] [Lincoln,
Lonnie Thompson] [rabbitmq][my_river71] failed to ack [498547]
com.rabbitmq.client.AlreadyClosedException: clean connection shutdown;
reason: Attempt to use closed channel
at
com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285)
at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872)
at
org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274)
at java.lang.Thread.run(Thread.java:636)

and then the river is not trying to reconnect (we waited at least 10
minutes). Is there a longer timeout or is this a bug?

Thank you for your great product!
Max

kimchy · June 12, 2012, 9:25am

Seems like the connection to rabbitmq is closed, wondering how it
happened... . Do you see any other messages?

On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz max.kossatz@gmail.com wrote:

Hi,

we noticed that when the RabbitMQ-Server fails the river is giving this
errormessage in the log:
[2012-06-08 15:45:14,362][WARN ][river.rabbitmq ] [Lincoln,
Lonnie Thompson] [rabbitmq][my_river71] failed to ack [498547]
com.rabbitmq.client.AlreadyClosedException: clean connection shutdown;
reason: Attempt to use closed channel
at
com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at
com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291)
at
com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285)
at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872)
at
org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274)
at java.lang.Thread.run(Thread.java:636)

and then the river is not trying to reconnect (we waited at least 10
minutes). Is there a longer timeout or is this a bug?

Thank you for your great product!
Max

dadoonet · June 12, 2012, 8:58pm

Hi Shay,

It's look really like what I have already seen with the twitter river.
After a failure, the river doesn't restart itself. And nothing in logs (only the failure with twitter).

I don't know if you remember this thread (it was in April I think), you asked me to set the river log level to debug but we did not see anything strange.

Could these issues be relative (Elasticsearch rivers internal) or not (relative to each river plugin) ?

Thanks
David
Twitter : @dadoonet / @elasticsearchfr

Le 12 juin 2012 à 11:25, Shay Banon kimchy@gmail.com a écrit :

Seems like the connection to rabbitmq is closed, wondering how it happened... . Do you see any other messages?

On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz max.kossatz@gmail.com wrote:
Hi,

we noticed that when the RabbitMQ-Server fails the river is giving this errormessage in the log:
[2012-06-08 15:45:14,362][WARN ][river.rabbitmq ] [Lincoln, Lonnie Thompson] [rabbitmq][my_river71] failed to ack [498547]
com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285)
at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872)
at org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274)
at java.lang.Thread.run(Thread.java:636)

and then the river is not trying to reconnect (we waited at least 10 minutes). Is there a longer timeout or is this a bug?

Thank you for your great product!
Max

Vladislav_Pernin · March 6, 2013, 3:08pm

Hi,

Has any of you found the solution ?

A thread dump shows that the river is not stopped but waits in the
LinkingBlockingQueue.

Maybe using a nextDelivery with a timeout to force to client to reconnect,
but I'm not sure that's the good solution ...

Vladislav

Le mardi 12 juin 2012 22:58:53 UTC+2, David Pilato a écrit :

Hi Shay,

It's look really like what I have already seen with the twitter river.
After a failure, the river doesn't restart itself. And nothing in logs
(only the failure with twitter).

I don't know if you remember this thread (it was in April I think), you
asked me to set the river log level to debug but we did not see anything
strange.

Could these issues be relative (Elasticsearch rivers internal) or not
(relative to each river plugin) ?

Thanks
David
Twitter : @dadoonet / @elasticsearchfr

Le 12 juin 2012 à 11:25, Shay Banon <kim...@gmail.com <javascript:>> a
écrit :

Seems like the connection to rabbitmq is closed, wondering how it
happened... . Do you see any other messages?

On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz <max.k...@gmail.com<javascript:>

wrote:

Hi,

we noticed that when the RabbitMQ-Server fails the river is giving this
errormessage in the log:
[2012-06-08 15:45:14,362][WARN ][river.rabbitmq ] [Lincoln,
Lonnie Thompson] [rabbitmq][my_river71] failed to ack [498547]
com.rabbitmq.client.AlreadyClosedException: clean connection shutdown;
reason: Attempt to use closed channel
at
com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at
com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291)
at
com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285)
at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872)
at
org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274)
at java.lang.Thread.run(Thread.java:636)

and then the river is not trying to reconnect (we waited at least 10
minutes). Is there a longer timeout or is this a bug?

Thank you for your great product!
Max

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Vladislav_Pernin · March 7, 2013, 9:07am

Hi,

After hours of debugging, I found
that consumer.nextDelivery(bulkTimeout.millis()) throws
a ShutdownSignalException.
This RuntimeException is not catched by the catch block which only catch
InterruptedException.
So the thread exit silently.

I filed a bug : RabbitMQ River : does not reconnect after RabbitMQ restart · Issue #2741 · elastic/elasticsearch · GitHub

Vlad

Le mercredi 6 mars 2013 16:08:15 UTC+1, Vladislav Pernin a écrit :

Hi,

Has any of you found the solution ?

A thread dump shows that the river is not stopped but waits in the
LinkingBlockingQueue.

Maybe using a nextDelivery with a timeout to force to client to reconnect,
but I'm not sure that's the good solution ...

Vladislav

Le mardi 12 juin 2012 22:58:53 UTC+2, David Pilato a écrit :

Hi Shay,

It's look really like what I have already seen with the twitter river.
After a failure, the river doesn't restart itself. And nothing in logs
(only the failure with twitter).

I don't know if you remember this thread (it was in April I think), you
asked me to set the river log level to debug but we did not see anything
strange.

Could these issues be relative (Elasticsearch rivers internal) or not
(relative to each river plugin) ?

Thanks
David
Twitter : @dadoonet / @elasticsearchfr

Le 12 juin 2012 à 11:25, Shay Banon kim...@gmail.com a écrit :

Seems like the connection to rabbitmq is closed, wondering how it
happened... . Do you see any other messages?

On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz max.k...@gmail.com wrote:

Hi,

we noticed that when the RabbitMQ-Server fails the river is giving this
errormessage in the log:
[2012-06-08 15:45:14,362][WARN ][river.rabbitmq ] [Lincoln,
Lonnie Thompson] [rabbitmq][my_river71] failed to ack [498547]
com.rabbitmq.client.AlreadyClosedException: clean connection shutdown;
reason: Attempt to use closed channel
at
com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at
com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291)
at
com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285)
at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872)
at
org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274)
at java.lang.Thread.run(Thread.java:636)

and then the river is not trying to reconnect (we waited at least 10
minutes). Is there a longer timeout or is this a bug?

Thank you for your great product!
Max

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · March 7, 2013, 9:12am

Very nice catch!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 7 mars 2013 à 10:07, Vladislav Pernin vladislav.pernin@gmail.com a écrit :

Hi,

After hours of debugging, I found that consumer.nextDelivery(bulkTimeout.millis()) throws a ShutdownSignalException.
This RuntimeException is not catched by the catch block which only catch InterruptedException.
So the thread exit silently.

I filed a bug : RabbitMQ River : does not reconnect after RabbitMQ restart · Issue #2741 · elastic/elasticsearch · GitHub

Vlad

Le mercredi 6 mars 2013 16:08:15 UTC+1, Vladislav Pernin a écrit :
Hi,

Has any of you found the solution ?

A thread dump shows that the river is not stopped but waits in the LinkingBlockingQueue.

Maybe using a nextDelivery with a timeout to force to client to reconnect, but I'm not sure that's the good solution ...

Vladislav

Le mardi 12 juin 2012 22:58:53 UTC+2, David Pilato a écrit :
Hi Shay,

It's look really like what I have already seen with the twitter river.
After a failure, the river doesn't restart itself. And nothing in logs (only the failure with twitter).

I don't know if you remember this thread (it was in April I think), you asked me to set the river log level to debug but we did not see anything strange.

Could these issues be relative (Elasticsearch rivers internal) or not (relative to each river plugin) ?

Thanks
David
Twitter : @dadoonet / @elasticsearchfr

Le 12 juin 2012 à 11:25, Shay Banon kim...@gmail.com a écrit :

Seems like the connection to rabbitmq is closed, wondering how it happened... . Do you see any other messages?

On Fri, Jun 8, 2012 at 6:13 PM, Max Kossatz max.k...@gmail.com wrote:
Hi,

we noticed that when the RabbitMQ-Server fails the river is giving this errormessage in the log:
[2012-06-08 15:45:14,362][WARN ][river.rabbitmq ] [Lincoln, Lonnie Thompson] [rabbitmq][my_river71] failed to ack [498547]
com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285)
at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:872)
at org.elasticsearch.river.rabbitmq.RabbitmqRiver$Consumer.run(RabbitmqRiver.java:274)
at java.lang.Thread.run(Thread.java:636)

and then the river is not trying to reconnect (we waited at least 10 minutes). Is there a longer timeout or is this a bug?

Thank you for your great product!
Max

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.