More consumer threads than the number specified in the input plugin

I have 6 logstash nodes. Each node has a pipeline with the following input section.

input {
  kafka {
    bootstrap_servers => "<list of kafka brokers>"
    client_id => "unique_client_id"
    group_id => "common_group_id_for_logstash"
    topics => [log_topic]
    tags => ["log"]
    security_protocol => "SSL"
    consumer_threads => 2
  }
}

The number of partitions is 12. So, each node should have 2 threads in the group and all threads should be listening to unique partitions. However, when I checked the consumer group in Kafka, I found the following distribution.

logstash01 - 4 threads
logstash02 - 4 threads
logstash06 - 4 threads

Why are more threads than 2 being used from the above hosts? Why are other hosts not getting any partitions allocated?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.