# Does Logstash's RabbitMQ output plugin produce messages in bulk?

**URL:** <https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368>\
**Category:** Logstash\
**Created:** [December 16, 2015, 3:28pm UTC](https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368 "2015-12-16T15:28:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sharon.c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sharon.c/32/16076_2.png) [@sharon.c](https://discuss.elastic.co/u/sharon.c)\
**Post date:** [December 16, 2015, 3:28pm UTC](https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368/1 "2015-12-16T15:28:30Z")

</div>

I am a new user for Logstash, and I want to use Logstash's RabbitMQ output plugin to feed data to RabbitMQ, the event speed I am processing is 3000-4000 events per second, I want to know if RabbitMQ output plugin is doing bulk process to publish the messages to RabbitMQ.

---

<div class="post-metadata">

**Author:** ![andrewvc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewvc/32/5064_2.png) [@andrewvc](https://discuss.elastic.co/u/andrewvc)\
**Post date:** [December 16, 2015, 3:55pm UTC](https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368/2 "2015-12-16T15:55:35Z")

</div>

Sharon, AFAIK batch processing in RabbitMQ is the application's responsibility. There is no RabbitMQ API that I'm aware of that lets one more efficiently enqueue multiple messages.

We could consider some sort of functionality that groups batches of events together and enqueues them as a single event on RabbitMQ however. Is that what you're thinking of?

---

<div class="post-metadata">

**Author:** ![sharon.c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sharon.c/32/16076_2.png) [@sharon.c](https://discuss.elastic.co/u/sharon.c)\
**Post date:** [December 16, 2015, 4:55pm UTC](https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368/3 "2015-12-16T16:55:09Z")

</div>

Yes, that is exactly what I am thinking of.

---

<div class="post-metadata">

**Author:** ![andrewvc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewvc/32/5064_2.png) [@andrewvc](https://discuss.elastic.co/u/andrewvc)\
**Post date:** [December 16, 2015, 8:39pm UTC](https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368/4 "2015-12-16T20:39:01Z")

</div>

We currently don't support that.

You might be able to increase throughput by increasing the workers setting in the rabbitmq section of your output

```
output { rabbitmq { workers => 4 } }

```

for instance would setup four separate worker threads.

If you'd like to add the batch functionality you describe we'd be gladly accept a patch that does that.

---

<div class="post-metadata">

**Author:** ![sharon.c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sharon.c/32/16076_2.png) [@sharon.c](https://discuss.elastic.co/u/sharon.c)\
**Post date:** [December 17, 2015, 4:10pm UTC](https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368/5 "2015-12-17T16:10:20Z")

</div>

Are the 4 worker threads using the same RabbitMQ connections?  
Another question, does Logstash RabbitMQ output plugin need to set up a connection, send message and then close the connection every time when a new message comes? Or it will set up a connection and keep it alive, and reconnect before the connection expires in every few minutes?

---

<div class="post-metadata">

**Author:** ![andrewvc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewvc/32/5064_2.png) [@andrewvc](https://discuss.elastic.co/u/andrewvc)\
**Post date:** [December 17, 2015, 6:35pm UTC](https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368/6 "2015-12-17T18:35:48Z")

</div>

Each worker will use its own connection. The connections are maintained between messages, so they will stay connected. You can also turn keepalive on.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:17am UTC](https://discuss.elastic.co/t/does-logstashs-rabbitmq-output-plugin-produce-messages-in-bulk/37368/7 "2017-07-06T05:17:58Z")

</div>

Closed on July 6, 2017, 5:17am UTC