Kafka increasing consumer lag

Elastic Stack Logstash
1

Hello! I have ELK stack installed with Kafka on Docker containers. There is lag between incoming logs and consuming logs that is increasing, and offset reading speed is not enough.
Kafka have only one topic. I tried different combinations of number of partitions in Kafka and pipelines in Logstash, also various combinations of batch.size, delay, consumer_threads parameters and it almost makes no difference.

Hardware is 6 cores CPU, memory is 350 Gb, RAM is 24 Gb, Java heap size is 12 Gb.

Logstash configuration for pipeline:


input {
  kafka {
    topics => "mytopic"
    bootstrap_servers => "bootstrap_server:9092"
    codec => "cef"
    auto_offset_reset => "latest"
    decorate_events => true
    consumer_threads => 6
    max_poll_records => 2000
    fetch_max_bytes => 209715200
    max_partition_fetch_bytes => 25165824
    fetch_max_wait_ms => 100
    session_timeout_ms => 600000
    heartbeat_interval_ms => 200000
  }
}
output {
    elasticsearch {
      ...
    }
  }

Logstash pipelines list:

- pipeline.id: pipe1
  path.config: "/usr/share/logstash/pipeline/logstash.conf"
  pipeline.batch.size: 500
  pipeline.batch.delay: 5
  pipeline.workers: 2

- pipeline.id: pipe2
  path.config: "/usr/share/logstash/pipeline/logstash.conf"
  pipeline.batch.size: 500
  pipeline.batch.delay: 5
  pipeline.workers: 2

- pipeline.id: pipe3
  path.config: "/usr/share/logstash/pipeline/logstash.conf"
  pipeline.batch.size: 500
  pipeline.batch.delay: 5
  pipeline.workers: 2

Kafka config is default with 6 partitions.

I want to get rid of lag on all partitions, how can I do that? I tried hundreds various configurations of parameters shown above, but lag is increasing anyway. Can it be issue with hardware?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.