Output to rabbitMQ does not honor reject-publish

Elastic Stack Logstash
1

Hi all,

I need help on rabbit output plugin.

We are pushing to rabbit with logstash.

For data integrity reasons, we want rabbit to apply back pressure on logstash. When rabbit is filling up (because our platform is unable to process the flow of data), it has two options for refusing data:

  • global watermarks (disk or memory), which block the whole broker until water level drops
  • per queue limits (on number of messages, or on data size), that block only a specific overwhelmed queue.

The first option (global watermarks) works well with the logstash output plugin: the back pressure is on logstash in this case. Fine. We are not fully satisfied with this solution since it blocks the whole broker and our platform uses the broker internally as a pipeline for data processing. In this case, all our data processing stops. (See description of the infra to the rabbit mq team for more details ).

We would like to do better with the "recent" per queue limits of rabbit (max-length, max-length-bytes). These block only the specified queue, and we use it with a reject-publish policy: the broker replies to publishers (logstash) with a negative ack ("rejected").

However, the logstash output plugin does not await publish confirms when pushing to rabbit. It does fail on connection errors, but with in our use case, rabbit just send a negative ack, which logstash ignores. And we lose the message.

I understand that logstash does not use publish confirms for performance reasons. In our use case, we can go with the performance penalty, if it gives us data integrity.

I guess this is linked with these posts:

I add that I will dig into the rabbitmq output plugin to see what I can do (logstash-integration-rabbitmq).

I appreciate any help on this issue:

  • am I missing something in logstash configurations?
  • is it linked with the above post ?
  • is it an open/abandoned issue in the roadmap of the logstash plugin?
  • have some people tried it apart from the above posts?
  • is it "easily" feasible?
  • I am not fluent on ruby, so any help on the structure of the plugin is appreciated: threads, mixin...

Thanks for reading!
Sylvain.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.