Logstash consumers from Kafka never increase beyond a certain count

Elastic Stack Logstash
1

Here is our config:

  • 64 replicas of Logstash Deployments (run ELK in Kubernetes cluster)
  • 64 partitions on the Kafka topic
  • 1 consumer thread per Logstash

Resources for Logstash:

  • Heap size: 3g
  • Cpu: 1000m (per logstash replica)
  • Memory: 4Gi

The problem is that Kafka consumers never go above 29. I expect it to go to 64 based on the above configuration.

Can someone help with this?

Thank you.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.