# Is it possible to do Load balancing using Kafka Input Plugin

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447>\
**Category:** Logstash\
**Created:** [January 5, 2024, 10:25am UTC](https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447 "2024-01-05T10:25:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Subrato1](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@Subrato1](https://discuss.elastic.co/u/Subrato1)\
**Post date:** [January 5, 2024, 10:25am UTC](https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447/1 "2024-01-05T10:25:33Z")

</div>

This is Configuartion I am Using.

```auto
input {
  kafka{
    #Insert any one string from the kafka_brokers_sasl from the service credential in the Event stream.
    bootstrap_servers => "<kafka_brokers_sasl>"
    #Insert the topic name which you created in Event stream.
    topics => "<topic_name>"
    security_protocol => "SASL_SSL"
    sasl_mechanism => "PLAIN"
    #Insert the username and password from the service credential in the Event stream.
    sasl_jaas_config => "org.apache.kafka.common.security.plain.PlainLoginModule required username='<user>' password='<password>';"
    type =>"icd_postgresql" 
    consumer_threads => 1
}

```

I want to use this Conf in two machine and I want my data should go to both machine but it only captured in one machine.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 5, 2024, 12:12pm UTC](https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447/2 "2024-01-05T12:12:52Z")

</div>

> [@Subrato1](#):
>
> it only captured in one machine.

Can you provide more context about this? Just one machine read from Kafka? How many partitions does your Kafka topic have?

If you use this configuration in multiple logstash, they would consume data from your kafka topic in parallel, I would also recomend to explicitly setting the `group_id` to some name instead of just relying on the default `group_id` name.

---

<div class="post-metadata">

**Author:** ![Subrato1](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@Subrato1](https://discuss.elastic.co/u/Subrato1)\
**Post date:** [January 8, 2024, 7:09am UTC](https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447/3 "2024-01-08T07:09:03Z")

</div>

I configured 1 partition for my Kafka Topic. I am using two different machines where I am using Different Broker SSL in both machines, but My data is Captured in only one machine.

GOAL: I want my data should go to both machines.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 8, 2024, 12:28pm UTC](https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447/4 "2024-01-08T12:28:21Z")

</div>

> [@Subrato1](#):
>
> I configured 1 partition for my Kafka Topic.

You need at least 2 partitions, with 1 partiton only one consumer will be able to consume from your topic, which is what you described.

---

<div class="post-metadata">

**Author:** ![Subrato1](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@Subrato1](https://discuss.elastic.co/u/Subrato1)\
**Post date:** [January 9, 2024, 4:36am UTC](https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447/5 "2024-01-09T04:36:17Z")

</div>

My Kafka has 6 Broker SSL so should I have to use all of them at once in both machines, or anyone would be sufficient for both machines??

Does Broker SSL somehow affect our incoming events?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 9, 2024, 12:22pm UTC](https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447/6 "2024-01-09T12:22:26Z")

</div>

> [@Subrato1](#):
>
> My Kafka has 6 Broker SSL so should I have to use all of them at once in both machines, or anyone would be sufficient for both machines??

It is not clear what you are asking.

Your kafka cluster has 6 brokers? In the configuration you can put them all or just a couple of them, it makes no difference as Kafka client always get a list of available brokers when connecting.

> [@Subrato1](#):
>
> Does Broker SSL somehow affect our incoming events?

I don't think so.

The issue is, if your topic has just 1 partition, only 1 consumer per log group will be able to consume its logs, since you have 2 logstash with the same `group_id`, one of them is idle, you need to increase the number of partitions according to the number of Logstash nodes you have.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 6, 2024, 12:23pm UTC](https://discuss.elastic.co/t/is-it-possible-to-do-load-balancing-using-kafka-input-plugin/350447/7 "2024-02-06T12:23:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.