RabbitMQ output is missing queue parameter

Elastic Stack Logstash
1

Hi,

The RabbitMQ output plugin is missing the queue parameter hence it can't create queues.
This makes any serious RabbitMQ usage impossible.

When writing to AMQP it should be possible to do 3 things:

  • create exchange (this works)
  • create queue (no option for that)
  • create a binding between exchange and queue using the provided key (key is ignored in fanout types)

I suggest to add another parameter called "queue" which defines the name of the queue which has to be created. The default could be "logstash"

The key parameter then defines the key used to bind the newly created (or already existing) queue to the newly created (or already existing) exchange.

Together with this you might also add following params::

  • queue durable (yes/no) default yes
  • queue auto delete (yes/no) default no

I suspect that the creator of the plugin has tried to simplify AMQP configuration but it's not how a good AMQP citizen should behave...

2

The RabbitMQ output plugin is missing the queue parameter hence it can't create queues.
This makes any serious RabbitMQ usage impossible.

Why should Logstash as a producer be responsible for creating queues? I don't think that makes sense at all. Queue creation is up to the consumers.

3

Why should Logstash as a producer be responsible for creating queues?

Because if your producers are started before your consumers then you would loose data since the consumers haven't created the queue yet to which data will arrive?

The last thing you'd want to do is to coordinate whoever is allowed to connect first to RabbitMQ ... The idea of AMQP is that producers and consumers aren't aware of each other.

It's not that a producer is responsible "per se" for creating queues. If you let both producing and consuming parties create the exchange, queue and queue binding then no data gets lost.

If the consumer is first to connect and create the exchange/queue/binding great, then all is ready for the producer to start dumping data into RabbitMQ.

If the producer is first to connect and create the exchange/queue/binding, then it can already start to produce data, which ends up in the created queue from which the consumer then can start to consume when its started and connected. No data lost.
If you submit data into an exchange to which no queue is bound then you loose data.

The precondition is that the consumer and producer create the exchange/queue/binding with exactly the same properties. Otherwise both parties have different expectations which is not good. RabbitMQ will then refuse the deviating exchange/queue/binding from being created with error "precondition failed".

This is how AMQP clients (producers and consumers) should behave.

4

The last thing you'd want to do is to coordinate whoever is allowed to connect first to RabbitMQ ... The idea of AMQP is that producers and consumers aren't aware of each other.

And yet producers are expected to know about all consumers and how their queues should be configured?

Anyway, feel free to file a feature request on GitHub. Requests made here aren't likely to be picked up.

5

And yet producers are expected to know about all consumers and how their queues should be configured?

No? Producers and consumers don't care how many are participating, where they're all connecting from, behind nat or not, how many are joining, leaving, ...

What needs to be agreed upon beforehand are the properties of how data is handled. Durable, TTL, acknowledgements, ... both the producer configuration and consumer configuration should be aligned for that.

Anyway, feel free to file a feature request on GitHub. Requests made here aren't likely to be picked up.

yeah did that first but I followed the suggestion made when creating a new issue:

Please post all product and debugging questions on our [forum](https://discuss.elastic.co/c/logstash). Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.