Logstash RabbitMQ Input Plugin Consumes Messages Despite ack => false Setting

Elastic Stack Logstash
1

Description:
I'm using Logstash with RabbitMQ input plugin to read messages from a queue and write them to Elasticsearch. I want Logstash to only read the messages without consuming them (like a logger), but it's still consuming the messages from the queue.

Current Configuration:

input {
  rabbitmq {
    host => "rabbitmq"
    port => 5672
    user => "admin"
    password => "admin"
    queue => "logstash_queue"
    key => "logstash_key"
    prefetch_count => 1
    durable => true
    ack => false
    auto_delete => false
    exchange => "logstash_exchange"
  }
}

Expected Behavior:

  • Logstash should only read messages from RabbitMQ queue
  • Messages should remain in the queue after being read
  • Other consumers should be able to read the same messages
  • Logstash should act as a logger, not a consumer

Actual Behavior:

  • Logstash is consuming messages from the queue
  • Messages are being removed from the queue after being read
  • Other consumers cannot read the same messages
  • Setting ack => false is not preventing message consumption

Environment:

  • Logstash Version: [8.12.1]
  • RabbitMQ Version: [rabbitmq:3.13-management]

Steps to Reproduce:

  1. Configure Logstash with the above configuration
  2. Send messages to RabbitMQ queue
  3. Observe that messages are being consumed from the queue
  4. Check that other consumers cannot read the same messages

Additional Information:

  • This behavior is critical for our logging system where we need to preserve messages in the queue for other consumers
  • We've tried various configurations including different exchange types and queue settings
  • The issue persists regardless of the ack setting

Question:
Is this the expected behavior of the RabbitMQ input plugin? If not, how can we configure Logstash to only read messages without consuming them?

2

I believe so. Setting the ack option on the input determines the manual_ack setting when the input subscribes. If the ack option is true then the input will ack the message, if it is false then I think the March Hare library will auto-ack the message for you.

1 Like
3

Thank you for your response. I've tried setting ack => true in my configuration, but the messages are still being consumed and removed from the queue. This behavior is not what we need since we want Logstash to only read the messages without consuming them, allowing other consumers to access the same messages.

Could you please clarify:

  1. Is there any other configuration setting we need to adjust to prevent message consumption?
  2. Are there any known limitations or workarounds for this behavior?
  3. Would using a different exchange type or queue configuration help achieve our goal of message preservation?

Our current configuration:

input {
  rabbitmq {
    host => "rabbitmq"
    port => 5672
    user => "admin"
    password => "admin"
    queue => "mirror_logstash_queue"
    key => "logstash_key"
    prefetch_count => 1
    durable => true
    ack => true
    auto_delete => false
    exchange => "logstash_exchange"
  }
}
4

Not that I know of. It looks like the input always acks the message one way or another.

My guess is you would need a fan-out exchange to create a queue for each consumer.

1 Like
5

I may be wrong, but I don't think that RabbitMQ works this way, if you have a consumer for a queue the messages will be consumed and removed from the queue, if you do not want this you need to do what Badger mentioned, which is basically duplicating your messages.

The behavior you want is more related to how Kafka works, where you can have a topic and multiple consumers reading from the topic without impacting each other if they use different group ids.

1 Like
6

Maybe - suggestion without knowing if it's possible - create a read only user on the rabbitmq side for using in logstash, will do it?
But like this, how do you know if a message was previously costumed? The messages will be on the queue forever (if nobody removed it)...

Good luck, Pedro

1 Like
7

... exactly what I was wondering. Something needs to keep track, or messages can be read 0, 1, or ... N times.

The terminology here doesn't help - reading a message != consuming a message? Is that really a widely understood industry standard distinction (in message queueing circles)?

8

It was an interesting approach :grinning_face_with_smiling_eyes:, it might have worked. However, I realized that reading messages without consuming them caused duplicates in some cases. So, I mirrored all the messages to a different queue and connected Logstash to that queue, which solved my problem. Thanks.

1 Like