Log stash rabbitmq cluster failing

Elastic Stack Logstash
1

I have a 3 node cluster of Rabbitmq behind a HAproxy Load Balancer. When I shut down a node, Rabbitmq successfully switches the queue to the other nodes. However, I notice that Logstash stops pulling messages from the queue unless I restart it. Is this a problem with the way rabbitmq operates? i.e. it deactivates all active consumers. I am not sure if log stash has any retry capability. Anyone run into this issue?

2

Logstash should clearly try to reconnect to another node. What's in the Logstash logs when this happens?

3

Which plugin version are you using, I had a pull request which fixed this error - try updating the plugin.

4

We use the Rabbitmq Java library's automatic connection recovery logic. So it seems the issue is buried somewhere in there. Any interest in providing a patch for https://github.com/logstash-plugins/logstash-input-rabbitmq ?

5

Thanks for your response. I turned off logging to stop the app from taking up too much disk space. Unfortunately, I shall turn debugging on and see what happens. What is the default behavior?

6

Thanks for your response. logstash-output-rabbitmq (3.0.7). It is a bit behind but not by much.

7

What is the default behavior?

That reconnection works without problems. Or what do you mean?

logstash-output-rabbitmq (3.0.7). It is a bit behind but not by much.

I suggest you try upgrading. It might not help, but any corrections are going to be based on the latest version anyway so there's no point in lagging behind.

8

Certainly would be an interesting exercise. Is this a defect? Or am I the first?

9

I mean that the is the default behavior of logstash to retry unack'd messages?

10

I mean that the is the default behavior of logstash to retry unack'd messages?

Yes.

11

I looked at the documentation and there are 2 properties
subscription_retry_interval_seconds and
connect_retry_interval

The documentation does not elaborate. Can you explain if these need to be set?

12

They have reasonable default values so you don't have to set them.

@Andrew_Cholakian1, can you look into why the rendered Asciidoc only includes the option descriptions for some options?

13

Thanks

14

Does not seem to be the case. The unacked messages still stay out there and unless I restart log stash , it does not pick up from the new primary rabbit node.