Hi,
My use case is
- I have multiple kafka topics, each with one partition. Say I have n topics, all topics names are like foo-*(can be defined by standard regex)
- I want multiple logstash instances to consume from these n topics. Say k logstash instances consuming from n topics
What happens when i add one more topic following the same regex, do the logstash consumers balance themselves accordingly. Also if I add one more logstash instance , will that also cause re-balancing among the partitions and consumers.
Basically what i want to know is, in case of addition of a new topic or a new logstash instance, will the load be re-distributed, or is it that the first logstash instance that was ever started will always be consuming from each partition as in my case all topics have 1 partition, and hence at a topic level the consumer:partition ratio will always be 1:1.