Hi!
I have a 6 node cluster running kafka broker on 3 machines and a topic with 18 partitions.
I started logstash process on 3 out of six nodes with a common consumer group subscribing to this topic.
But when I describe consumer-group:
/bin/kafka-consumer-groups.sh --bootstrap-server : --group --describe
It shows a same logstash owner for all 18 partitions. Shouldn't I see the partitions divided among 3 logstash processes that I have running on 2 nodes. Rightnow looks like only one logstash consumer is working.
Plus from zkCli.sh ls /consumers doesn't show
Restarting ZK and kafka doesn't help.
Here is my logstash input section:
input {
kafka {
bootstrap_servers => "ip-...internal:6667,ip-.internal:6667"
group_id => "my-group"
topics => ["input"]
consumer_threads => 2
}
}