How many consumer_threads of Kafka input to configure if duplicate Logstash instances

Elastic Stack Logstash
1

Hi,

I am using ELK GA 6.3.0. I am consuming from a Kafka topic which has 11 partitions. My configuration is like below;

input {
	kafka{
		group_id => "group1"
		topics => ["topic1"]
		bootstrap_servers => "192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092"
		codec => "json"
	}
}

In the plugin doc, it is mentioned that;

Ideally you should have as many threads as the number of partitions for a perfect balance — more threads than partitions means that some threads will be idle

So I changed my configuration like this;

input {
	kafka{
		group_id => "group1"
		topics => ["topic1"]
		bootstrap_servers => "192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092"
		codec => "json"
		consumer_threads => "11"
	}
}

I can understand this if I have one instance of Logstash. But I am running 2 Logstash instances. My intentions are;

  1. Consume from topic even if one Logstash terminates / fails
  2. Load distribution.

So my question is, how many consumer_threads should I give - is it 11 for both or 5 for one and 6 for other ?

Thanks.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.