Increased Kafka Lag due to load balancing

Elastic Stack Logstash
1

Hello,

I am running logstash 2.2.2 and I recently upgraded kafka input/output 3.x plugin to support consumer offset storage to Kafka 0.9
As a result, I am connecting to brokers as oppose to Zookeepers. But what I have noticed, since this change, the load balancing is broken. As not all consumers are being assigned to topic partitions resulting in kafka lag increase. Before the upgrade i didn't have this issue as zookeepers where handling the loadbalancing

2

I haven't run this setup yet so I don't have much good advice other than you might want to check the Kafka user mailing lists http://grokbase.com/g/kafka/users

Are all topics being assigned to a consumer or are some being left out? I'm trying to understand if the problem is that the topics aren't being distributed evenly or not at all.