More consumer threads than the number specified in the input plugin

Elastic Stack Logstash
1

I have 6 logstash nodes. Each node has a pipeline with the following input section.

input {
  kafka {
    bootstrap_servers => "<list of kafka brokers>"
    client_id => "unique_client_id"
    group_id => "common_group_id_for_logstash"
    topics => [log_topic]
    tags => ["log"]
    security_protocol => "SSL"
    consumer_threads => 2
  }
}

The number of partitions is 12. So, each node should have 2 threads in the group and all threads should be listening to unique partitions. However, when I checked the consumer group in Kafka, I found the following distribution.

logstash01 - 4 threads
logstash02 - 4 threads
logstash06 - 4 threads

Why are more threads than 2 being used from the above hosts? Why are other hosts not getting any partitions allocated?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.