Kafka consumer_threads

Elastic Stack Logstash
1

Hi all,

We have 1 Kafka topic with 3 partitions and 2 Logstash (each one running on it own linux vm), that we can call logstas-A
and logstash-B, both having same pipeline configured. Pior to this configuration we had one Logstash consuming all 3
partitions but it started to show some LAG, so we join a second Logastash.

Logstash-A is configured with 2 consumer_threads and logstash-B with just one.

Questions:

  1. Is this consumer_threads configuration OK or should we have both logstash configured with 3 consumer_threads?

  2. If both logstash are configured with 3 consumer_threads we may end with one Logstash consuming from all 3 partitions, right? (which we do not want because performance)

  3. In case one Logastash stop for maintenance what happens? Will the other Logasth start consuming from the "abandoned" partition?

  4. What happens when stoped logstash is back again? Will it consume any partition at all?

  5. In this case what is the best consumer_threads configuration to have both load distribution and resilience to fail/stop?

Logstash v8.10

Thank you.