Logstash Kakfa is Slow and Rebalancing Often

[dhinesh]

Hi.,

I have logstash reading from Kafka topic with 3 paritions. The logstash is slow in processing the data and kafka is rebalancing very often, which is causing duplication of records. I have tried to increase the max_poll_timeout =600000ms and session_timeout = 300000ms with no luck. I need some guidance to solve the slowness problem.

    **Logstash config:**

    input {
      kafka {
        bootstrap_servers => '1.1.1.1:8989'
        group_id => 'Test'
        topics => 'Topic1'
        codec => json
        max_poll_records => '50'
       }
    }

    filter {
      ruby {
        code => "
          tstamp = event.get('epoch')
          event.set('Time',Time.at(tstamp.to_i).strftime('%Y-%m-%d %H:%M:%S'))
          tstamp.clear
        "
      }
    }

    output {

      stdout { codec => rubydebug  {  metadata => true } }

      csv {
        file_mode => 0644
       
        fields => ['Time','Field1','Field2','Field3']
        path => '/csv/data.csv'
      }
      
      }
    }

    **Logstah.yaml**

    http.host: "0.0.0.0"
    log.level: info
    path.logs: /logs
    pipeline.workers: 3
    pipeline.batch.size: 10000
    config.reload.automatic: "true"
    config.reload.interval: "3s"
    path.data: /data