# Logstash not pulling data fast enough from Kafka

**URL:** <https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377>\
**Category:** Logstash\
**Created:** [May 26, 2023, 12:05am UTC](https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377 "2023-05-26T00:05:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Francisco\_Yanez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francisco_yanez/32/92021_2.png) [@Francisco\_Yanez](https://discuss.elastic.co/u/Francisco_Yanez)\
**Post date:** [May 26, 2023, 12:05am UTC](https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377/1 "2023-05-26T00:05:06Z")

</div>

I have a huge problem. My kafka is on a different DC and we are using logstash to pull data. Our Elastic stack is running in kubernetes but our data is getting pulled very slow. How can I optimize logstash to pull data faster?

---

<div class="post-metadata">

**Author:** ![X11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/x11/32/98759_2.png) [@X11](https://discuss.elastic.co/u/X11)\
**Post date:** [May 27, 2023, 6:10am UTC](https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377/2 "2023-05-27T06:10:09Z")

</div>

How do you know the bottleneck is logstash? How many logstash consumers do you have reading from the Kafka topic ? How many partitions does your Kafka topic have ? What does your Kafka input configuration look like at the logstash side ? What’s the spec of the the logstash consumer ?

---

<div class="post-metadata">

**Author:** ![Francisco\_Yanez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francisco_yanez/32/92021_2.png) [@Francisco\_Yanez](https://discuss.elastic.co/u/Francisco_Yanez)\
**Post date:** [May 27, 2023, 9:14pm UTC](https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377/3 "2023-05-27T21:14:29Z")

</div>

thanks for the help! let me answer those questions here:  
How do you know the bottleneck is logstash?  
I have installed redpanda on my cloud and I can see the lag on my kafka broker. So I see a huge lag, quite often and it takes days to clear.

How many logstash consumers do you have reading from the Kafka topic ?  
I have 12 partitions, so I have 12 logstash input kafka configured to pull from kafka as consumers

What does your Kafka input configuration look like at the logstash side ?  
producer is logstash output but I do see the messages in redpanda almost in realtime so this part is good.

What’s the spec of the the logstash consumer ?

```auto
input {
      kafka {
        bootstrap_servers => "my-server-kafka"
        topics => ["logstash"]
        codec => "json"
        group_id => "logstash"
        auto_offset_reset => "latest"
        session_timeout_ms => "250000"
        request_timeout_ms => "300000"
        security_protocol => "SSL"
        ssl_endpoint_identification_algorithm => ""
        ssl_keystore_location => "/usr/share/logstash/keystore/keystore"
        ssl_key_password => "Password"
        ssl_keystore_password => "Password"
        ssl_truststore_location => "/usr/share/logstash/keystore/truststore"
        ssl_truststore_password => "Password"
        fetch_max_wait_ms => "500"
        fetch_max_bytes => "96582912"
        fetch_min_bytes => "6048576"
        max_partition_fetch_bytes => "8048576"
        consumer_threads => "12"
        max_poll_records => "2000"
      }

```

---

<div class="post-metadata">

**Author:** ![Francisco\_Yanez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/francisco_yanez/32/92021_2.png) [@Francisco\_Yanez](https://discuss.elastic.co/u/Francisco_Yanez)\
**Post date:** [June 6, 2023, 10:43pm UTC](https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377/4 "2023-06-06T22:43:48Z")

</div>

I fixed this but in case anyone has any problems with logstash input as kafka consumer this is a great article:

> **[Kafka input plugin | Logstash Reference \[8.8\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-max_poll_records)**

and this is my config now:

```auto
input {
      kafka {
        bootstrap_servers => "my-server-kafka"
        topics => ["logstash"]
        codec => "json"
        group_id => "logstash"
        auto_offset_reset => "latest"
        session_timeout_ms => "250000"
        request_timeout_ms => "300000"
        security_protocol => "SSL"
        ssl_endpoint_identification_algorithm => ""
        ssl_keystore_location => "/usr/share/logstash/keystore/keystore"
        ssl_key_password => "Password"
        ssl_keystore_password => "Password"
        ssl_truststore_location => "/usr/share/logstash/keystore/truststore"
        ssl_truststore_password => "Password"
        fetch_max_wait_ms => 3000
        fetch_max_bytes => "96582912"
        fetch_min_bytes => "6048576"
        max_partition_fetch_bytes => "8048576"
        consumer_threads => "12"
        max_poll_records => "2000"
      }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 4, 2023, 10:44pm UTC](https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377/5 "2023-07-04T22:44:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.