# Using RabbitMq as broker between Beats and Logstash **URL:** \ **Category:** Beats\ **Created:** [May 11, 2024, 11:36am UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302 "2024-05-11T11:36:05Z")\ **Posts on this page:** 9\ **Page:** 2
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 12, 2024, 1:21pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/21 "2024-05-12T13:21:29Z")
> [@NasrJBr](#): > > My question is, what is the best configuration to apply for a Kafka broker with Logstash in the context of a SIEM ? This depends on the volume of data you have and for how many time you want for this data to be available in the broker. For most of cases there is basically zero need for tuning Kafka or Logstash, just keep in mind to use more than one partition on Kafka side, ideally try to have the number of partitions equal to the number of logstash nodes. You need to start small and improve as needed, it is pretty common to see people trying to start with the optimal configuration for all tools, which is a mistake in my opinion. ---
**Author:** ![NasrJBr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nasrjbr/32/134034_2.png) [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)\ **Post date:** [May 12, 2024, 1:46pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/22 "2024-05-12T13:46:00Z")
I want to create some scenarios and choose the optimal one for my use case because I have some configurations, but expertise plays an important role for this type of configurations. ---
**Author:** ![NasrJBr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nasrjbr/32/134034_2.png) [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)\ **Post date:** [May 12, 2024, 1:47pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/23 "2024-05-12T13:47:47Z")
Ok, i will start by the number of partitions equal to the number of logstash nodes and see the performance. Thank you @leandrojmp ---
**Author:** ![intrepid1](https://avatars.discourse-cdn.com/v4/letter/i/dbc845/32.png) [@intrepid1](https://discuss.elastic.co/u/intrepid1)\ **Post date:** [May 16, 2024, 1:57pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/24 "2024-05-16T13:57:27Z")
Hi there, Have you thought about using Redis? We sit Redis between our Filebeat servers and Logstash instances. We have a bank of Redis single instances that we load balance over from Filebeat. We set-up our Logstash servers with the Redis Logstash input set-up to read from any of the Redis instances. For us it works really well and supports the ingestion of around 2 billion loglines a day. ---
**Author:** ![NasrJBr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nasrjbr/32/134034_2.png) [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)\ **Post date:** [May 17, 2024, 8:43am UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/25 "2024-05-17T08:43:47Z")
Hello @intrepid1, But is Redis scalable in case the system becomes bigger? Can you explain more about how you did it? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 17, 2024, 12:07pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/26 "2024-05-17T12:07:32Z")
Keep in mind that if you use the Elastic Agent you cannot send logs to Redis, it is not a supported output. With the Elastic Agent only Logstash, Elasticsearch and Kafka are supported outputs. ---
**Author:** ![intrepid1](https://avatars.discourse-cdn.com/v4/letter/i/dbc845/32.png) [@intrepid1](https://discuss.elastic.co/u/intrepid1)\ **Post date:** [May 17, 2024, 3:34pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/27 "2024-05-17T15:34:03Z")
This is how we configure it, but please bear in mind Leandro's last points. We use a bank of single instance Redis. In the example here, we have three Redis instances. We configure Filebeat to round robin to our Redis instances as follows: ```auto output.redis: hosts: ["server1","server2","server3"] ``` When we configure Logstash, is configured as follows: ```auto input { redis { host => "server1" port => "6379" codec => "json" data_type => "list" key => "list_name" threads => 8 } redis { host => "server2" port => "6379" codec => "json" data_type => "list" key => "list_name" threads => 8 } redis { host => "server3" port => "6379" codec => "json" data_type => "list" key => "list_name" threads => 8 } } ``` If you want to scale out, you introduce a new Redis instance and then update the Filebeat and Logstash configurations appropriately. ---
**Author:** ![NasrJBr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nasrjbr/32/134034_2.png) [@NasrJBr](https://discuss.elastic.co/u/NasrJBr)\ **Post date:** [May 19, 2024, 12:39pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/28 "2024-05-19T12:39:03Z")
Thank you @intrepid1. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 16, 2024, 12:39pm UTC](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302/29 "2024-06-16T12:39:36Z")
Closed on June 16, 2024, 12:39pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/using-rabbitmq-as-broker-between-beats-and-logstash/359302.md?page=1)