# Log stash rabbitmq cluster failing **URL:** \ **Category:** Logstash\ **Created:** [April 6, 2016, 9:40pm UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611 "2016-04-06T21:40:53Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![rubiconvale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rubiconvale/32/31389_2.png) [@rubiconvale](https://discuss.elastic.co/u/rubiconvale)\ **Post date:** [April 6, 2016, 9:40pm UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/1 "2016-04-06T21:40:53Z")
I have a 3 node cluster of Rabbitmq behind a HAproxy Load Balancer. When I shut down a node, Rabbitmq successfully switches the queue to the other nodes. However, I notice that Logstash stops pulling messages from the queue unless I restart it. Is this a problem with the way rabbitmq operates? i.e. it deactivates all active consumers. I am not sure if log stash has any retry capability. Anyone run into this issue? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 7, 2016, 5:50am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/2 "2016-04-07T05:50:14Z")
Logstash should clearly try to reconnect to another node. What's in the Logstash logs when this happens? ---
**Author:** ![Logan1](https://avatars.discourse-cdn.com/v4/letter/l/c77e96/32.png) [@Logan1](https://discuss.elastic.co/u/Logan1)\ **Post date:** [April 7, 2016, 11:02am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/3 "2016-04-07T11:02:22Z")
Which plugin version are you using, I had a pull request which fixed this error - try updating the plugin. ---
**Author:** ![Andrew\_Cholakian1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_cholakian1/32/3612_2.png) [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)\ **Post date:** [April 7, 2016, 4:16pm UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/4 "2016-04-07T16:16:04Z")
We use the Rabbitmq Java library's automatic connection recovery logic. So it seems the issue is buried somewhere in there. Any interest in providing a patch for [https://github.com/logstash-plugins/logstash-input-rabbitmq](https://github.com/logstash-plugins/logstash-input-rabbitmq) ? ---
**Author:** ![rubiconvale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rubiconvale/32/31389_2.png) [@rubiconvale](https://discuss.elastic.co/u/rubiconvale)\ **Post date:** [April 8, 2016, 5:45am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/5 "2016-04-08T05:45:00Z")
Thanks for your response. I turned off logging to stop the app from taking up too much disk space. Unfortunately, I shall turn debugging on and see what happens. What is the default behavior? ---
**Author:** ![rubiconvale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rubiconvale/32/31389_2.png) [@rubiconvale](https://discuss.elastic.co/u/rubiconvale)\ **Post date:** [April 8, 2016, 5:45am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/6 "2016-04-08T05:45:55Z")
Thanks for your response. logstash-output-rabbitmq (3.0.7). It is a bit behind but not by much. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 8, 2016, 5:47am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/7 "2016-04-08T05:47:32Z")
> What is the default behavior? That reconnection works without problems. Or what do you mean? > logstash-output-rabbitmq (3.0.7). It is a bit behind but not by much. I suggest you try upgrading. It might not help, but any corrections are going to be based on the latest version anyway so there's no point in lagging behind. ---
**Author:** ![rubiconvale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rubiconvale/32/31389_2.png) [@rubiconvale](https://discuss.elastic.co/u/rubiconvale)\ **Post date:** [April 8, 2016, 5:47am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/8 "2016-04-08T05:47:55Z")
Certainly would be an interesting exercise. Is this a defect? Or am I the first? ---
**Author:** ![rubiconvale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rubiconvale/32/31389_2.png) [@rubiconvale](https://discuss.elastic.co/u/rubiconvale)\ **Post date:** [April 8, 2016, 5:57am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/9 "2016-04-08T05:57:18Z")
I mean that the is the default behavior of logstash to retry unack'd messages? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 8, 2016, 6:00am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/10 "2016-04-08T06:00:56Z")
> I mean that the is the default behavior of logstash to retry unack'd messages? Yes. ---
**Author:** ![rubiconvale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rubiconvale/32/31389_2.png) [@rubiconvale](https://discuss.elastic.co/u/rubiconvale)\ **Post date:** [April 8, 2016, 6:10am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/11 "2016-04-08T06:10:55Z")
I looked at the documentation and there are 2 properties subscription\_retry\_interval\_seconds and connect\_retry\_interval The documentation does not elaborate. Can you explain if these need to be set? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 8, 2016, 6:14am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/12 "2016-04-08T06:14:34Z")
They have reasonable default values so you don't have to set them. @Andrew_Cholakian1, can you look into why the rendered Asciidoc only includes the option descriptions for some options? ---
**Author:** ![rubiconvale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rubiconvale/32/31389_2.png) [@rubiconvale](https://discuss.elastic.co/u/rubiconvale)\ **Post date:** [April 8, 2016, 6:17am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/13 "2016-04-08T06:17:10Z")
Thanks ---
**Author:** ![rubiconvale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rubiconvale/32/31389_2.png) [@rubiconvale](https://discuss.elastic.co/u/rubiconvale)\ **Post date:** [April 11, 2016, 11:50pm UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/14 "2016-04-11T23:50:38Z")
Does not seem to be the case. The unacked messages still stay out there and unless I restart log stash , it does not pick up from the new primary rabbit node. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:02am UTC](https://discuss.elastic.co/t/log-stash-rabbitmq-cluster-failing/46611/15 "2017-07-06T05:02:53Z")