# Using RabbitMq as broker between Beats and Logstash
**URL:** https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302
**Category:** Beats
**Created:** [May 11, 2024, 11:36am UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302 "2024-05-11T11:36:05Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 11, 2024, 11:36am UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/1 "2024-05-11T11:36:05Z")
Hello, I’ve created an architecture for a SIEM using ELK. In this architecture, I’ve used a RabbitMQ broker between Beats and Logstash. Is it a good choice? My final target is to collect logs from at least 1000 Beats and forward them through RabbitMQ to be processed by Logstash. The use of the broker in this architecture is to bypass the problem of bottlenecking between the Beats and Logstash in case of too many logs. I chose RabbitMQ instead of Kafka because I only need to forward logs from Beats to Logstash and I wanted to avoid the complexity of Kafka deployment. My question is, is it a good choice to use RabbitMQ in this scenario? What are your recommendations?
Thank you
---
### Author:  [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)
#### Post date: [May 11, 2024, 1:05pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/2 "2024-05-11T13:05:44Z")
Hi @NasrJBr,
You can directly use logstash in filebeat's output. No need to use rabbitmq unless there is specific need. Logstash will also queued up all the events persistently.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 11, 2024, 1:09pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/3 "2024-05-11T13:09:54Z")
As far as I know Beats [are not able to output directly to RabbitMQ](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html) so I do not think this will work. Either deploy Kafka or send it directly to Logstash. Logstash is able to enqueue on disk, but as it is local disk it does not give you the resiliency that Kafka would offer.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [May 11, 2024, 1:57pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/4 "2024-05-11T13:57:28Z")
> [@NasrJBr](#):
>
> My question is, is it a good choice to use RabbitMQ in this scenario? What are your recommendations?
I would say that it is not, mainly because beats cannot output data directly to rabbitmq, so you would need another piece to manage in your infrastructure.
Kafka is a way better choice as you can output from beats directly into a Kafka topic.
I've been using Kafka as a message broker in ELK deployments for so many years that every time I need to spin up a new Elastic Cluster I consider Kafka as an essential part of it.
---
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 11, 2024, 2:38pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/5 "2024-05-11T14:38:00Z")
Hi @ashishtiwari1993, won’t there be a risk of data loss from Beats if there’s a connection or performance issue?
---
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 11, 2024, 2:39pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/6 "2024-05-11T14:39:15Z")
Hi @Christian_Dahlqvist, I believe RabbitMQ is mentioned in the official documentation.
---
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 11, 2024, 2:43pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/7 "2024-05-11T14:43:17Z")
Hi @leandrojmp, okay, so using Kafka is better than RabbitMQ for handling these types of problems. Initially, I considered using direct ingestion, but I took into consideration potential bottlenecks and data loss in case of problems, so that i add broker to my architecture.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 11, 2024, 2:43pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/8 "2024-05-11T14:43:29Z")
The link I provided shows the supported outputs, and RabbitMQ is not on it. There is however a module for collecting logs (possibly also metrics) from RabbitMQ, but that is very different.
---
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 11, 2024, 2:50pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/9 "2024-05-11T14:50:27Z")
Ah, okay. So there is another approach for adding RabbitMQ. From your perspective, you recommend using Kafka instead of RabbitMQ because my final objective is to eliminate bottlenecks and data loss.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 11, 2024, 2:53pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/10 "2024-05-11T14:53:14Z")
Yes, I would recommend using Kafka as I have seen it used successfully in deployments with very high throughput numbers. I am not sure how RabbitMQ performance compares, but believe it is significantly slower.
---
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 11, 2024, 3:00pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/11 "2024-05-11T15:00:25Z")
Ok, thank you. One more thing, is there any other alternative to be use instead of a broker to handle the two problems (bottlenecks and data loss)
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 11, 2024, 3:01pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/12 "2024-05-11T15:01:49Z")
No, I think using Kafka is the best option. It is a very common pattern and therefore asy to get help around.
---
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 11, 2024, 3:03pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/13 "2024-05-11T15:03:22Z")
ok, thank you for you help @Christian_Dahlqvist.
---
### Author:  [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)
#### Post date: [May 11, 2024, 4:42pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/14 "2024-05-11T16:42:33Z")
Filebeat guarantees that events will be delivered to the configured output at least once and with no data loss. Filebeat is able to achieve this behavior because it stores the delivery state of each event in the registry file.
Once data delivered to logstash, logstash will write to the disk. So there is no data loss.
In case of hight traffic, you can add more logstash server behind the load balancer.
---
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 11, 2024, 4:49pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/15 "2024-05-11T16:49:18Z")
Hi @ashishtiwari1993, instead of using a broker, can I use a load balancer between Beats and Logstash? Do all Beats support data loss prevention like Filebeat? Also, can you recommend some load balancers for this SIEM use case?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 11, 2024, 4:57pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/16 "2024-05-11T16:57:44Z")
> [@ashishtiwari1993](#):
>
> Once data delivered to logstash, logstash will write to the disk. So there is no data loss.
When you write data to Kafka, the data is distributed across the cluster and the los of a single node does not lead to data loss. Logstash only persist data to the local disk, so losing a Logstash node would likely result in data loss.
Another benefit of using Kafka is that it helps distribute the processing load evenly across processes pulling from it. With a large number of Logstash instances you do run the risk of having the Logstash instances very unevenly loaded.
Using Kafka is therefore in my opinion the superior option.
---
### Author:  [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)
#### Post date: [May 11, 2024, 5:03pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/17 "2024-05-11T17:03:28Z")
Agreed!!! For distributed architecture, kafka is the best. Just curious, distributing same events can led to data duplication (If we're pushing to Elasticsearch, same cluster.)
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [May 11, 2024, 5:37pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/18 "2024-05-11T17:37:13Z")
> [@ashishtiwari1993](#):
>
> Just curious, distributing same events can led to data duplication (If we're pushing to Elasticsearch, same cluster.)
It depends on how they are distributed, using Kafka this is not an issue as you can use the same group id for each logstash, then the events will not be duplicated.
A good approach is to have the number of partitions on Kafka to be the same as the number of Logstash nodes, this way each logstash node will consume from one partition and the events will be evenly distributed.
---
### Author:  [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)
#### Post date: [May 12, 2024, 12:03pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/19 "2024-05-12T12:03:57Z")
When we talk about data loss prevention using a load balancer, it can be inefficient due to overloads from Beats agents. Therefore, the best solution is to use a broker. My question is, what is the best configuration to apply for a Kafka broker with Logstash in the context of a SIEM ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 12, 2024, 1:17pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/20 "2024-05-12T13:17:55Z")
> [@NasrJBr](#):
>
> My question is, what is the best configuration to apply for a Kafka broker with Logstash in the context of a SIEM ?
Why would this be different? You are still shipping data and want to do so in a performant and reliable fashion.
[Next page](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302.md?page=2)