Hi,
Continuing the discussion from Multiple logstash reading from a single kafka topic:
I have 6 Kafka Instances running. I created a topic with 18 Partitions and 3 as a replication factor.
On the other hand I have 6 logstash Instances running for consuming messages from the topic.
When I tried to describe the consumer group offsets using the below command, I noticed that the consumer groups are always rebalancing and the partitions are not equally distributed among the logstash Instances. For example logstash_/192.168.188.8 has more load to process.
I used the following configuration in kafka input plugin.
consumer_threads => 3
** auto_offset_reset => "earliest"**
Should I increase the consumer_threads in order to avoid the backlog and distribute the load equally? Also I noticed that rebalancing is taking place almost every minute and the ownership of the partition is always changed. Is this optimal behavior? Why are the owners of the partitions changed so often?
./kafka-consumer-groups.sh --new-consumer --describe --bootstrap-server kafka:9092 --group logstash
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
logstash logdata 9 16946718 19586295 2639577 logstash_/192.168.172.9
logstash logdata 13 18078100 19598964 1520864 logstash_/192.168.188.8
logstash logdata 12 18515986 19583343 1067357 logstash_/192.168.151.5
logstash logdata 0 16865627 19595982 2730355 logstash_/192.168.188.8
logstash logdata 1 15847887 19582710 3734823 logstash_/192.168.188.8
logstash logdata 8 15754769 17880248 2125479 logstash_/192.168.151.5
logstash logdata 14 16980307 17880254 899947 logstash_/192.168.136.6
logstash logdata 17 18936788 19586336 649548 logstash_/192.168.188.5
logstash logdata 2 16440360 19570173 3129813 logstash_/192.168.188.8
logstash logdata 3 16206010 19586306 3380296 logstash_/192.168.188.8
logstash logdata 10 14811653 19595993 4784340 logstash_/192.168.151.5
logstash logdata 7 16417739 19582715 3164976 logstash_/192.168.172.9
logstash logdata 5 18606618 19598954 992336 logstash_/192.168.163.7
logstash logdata 16 17517220 19582783 2065563 logstash_/192.168.188.5
logstash logdata 4 12916114 19582814 6666700 logstash_/192.168.172.9
logstash logdata 11 18793688 19598933 805245 logstash_/192.168.163.7
logstash logdata 15 18613052 19595878 982826 logstash_/192.168.136.6
logstash logdata 6 17437378 19582042 2144664 logstash_/192.168.136.6