Here is our config:
- 64 replicas of Logstash Deployments (run ELK in Kubernetes cluster)
- 64 partitions on the Kafka topic
- 1 consumer thread per Logstash
Resources for Logstash:
- Heap size: 3g
- Cpu: 1000m (per logstash replica)
- Memory: 4Gi
The problem is that Kafka consumers never go above 29. I expect it to go to 64 based on the above configuration.
Can someone help with this?