I have a really weird case where I have a 6 logstash consumers consuming from a kafka topic using the logstash-kafka-input plugin. The 6 logstash consumers are their own logstash instances on different machines (and have run with no issue for some time). They have the SAME consumer group, and the topic has 8 partitions.
In the past when I started up a group of logstash instances like this, I could watch the logs as they shifted around and each got assigned a partition or two. However, currently, when I restart these machines I am seeing behavior where the LATEST logstash instance that starts up with take over ALL of the available partitions, stealing them from the last instance that was using them.
this is seriously damaging our ability to consume from kafka using logstash. any thoughts on possible causes would be appreciated!
Using logstash 5.2.