# Kafka Output - Increasing batching reduces throughput **URL:** \ **Category:** Logstash\ **Created:** [May 6, 2020, 11:30pm UTC](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426 "2020-05-06T23:30:10Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![taylorn](https://avatars.discourse-cdn.com/v4/letter/t/c6cbf5/32.png) [@taylorn](https://discuss.elastic.co/u/taylorn)\ **Post date:** [May 6, 2020, 11:30pm UTC](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426/1 "2020-05-06T23:30:10Z")
I'm seeing poor performance with the Kafka output plugin when I increase the linger\_ms and batch\_size settings. This is the opposite behavior I would expect after reading the Kafka producer documentation. Increasing batch sizes should improve throughput, not reduce it! I have not seen this issue with other Kafka producer implementations, so it seems like an issue specific to Logstash. My pipeline is very simple - read TCP input, apply no filters, then send to Kafka output. Other notable settings are: - persistent queue feature enabled - 8 worker threads - running on VM with 4 CPUs, 8GB RAM Here are some sample results of just tweaking the linger\_ms setting: linger\_ms=250 batch\_size=65536 **output events per second=14000** linger\_ms=10000 batch\_size=65536 **output events per second=1200** I have enabled JMX metrics on my logstash instance to look at the Kafka producer statistics. From this I can see that increasing linger\_ms does result in larger sized batches being sent to Kafka, and that records queue for linger\_ms amount of time before being sent. But still, the performance is greatly reduced despite improved batching. ![Screen Shot 2020-05-06 at 4.23.33 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dc8559b73b10feebbde54c1a988819a265dfa26a.png) ![Screen Shot 2020-05-06 at 4.23.45 PM](https://us1.discourse-cdn.com/elastic/original/3X/9/7/97358b07f69617acc8d1657b3a96f98881e6ae8c.png) ![Screen Shot 2020-05-06 at 4.23.58 PM](https://us1.discourse-cdn.com/elastic/original/3X/5/0/50392e29174a9549cfb55cb7f2d52b1fc91775c4.png) ![Screen Shot 2020-05-06 at 4.24.23 PM](https://us1.discourse-cdn.com/elastic/original/3X/f/9/f9a14458c7147a54436560929c8c1ccfabdfb713.png) ![Screen Shot 2020-05-06 at 4.29.22 PM](https://us1.discourse-cdn.com/elastic/original/3X/4/0/4003e71e09570b76f583c6913a0c595307b91a69.png) How can I find the bottleneck for this pipeline? Are there known issues with the Kafka output plugin? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 7, 2020, 4:19pm UTC](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426/2 "2020-05-07T16:19:55Z")
> [@taylorn](#): > > linger\_ms=10000 Your batch\_size is 65536, but the actual batch sizes shown in JMX are around 6,000, so events are going to be delayed by 10 seconds, lingering to see if the batch can be filled. I would think the bottleneck for the pipeline is the amount of time the events linger! ---
**Author:** ![taylorn](https://avatars.discourse-cdn.com/v4/letter/t/c6cbf5/32.png) [@taylorn](https://discuss.elastic.co/u/taylorn)\ **Post date:** [May 7, 2020, 5:10pm UTC](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426/3 "2020-05-07T17:10:09Z")
Hi Badger, Thanks for the response. I would agree except that my persistent queue is filling up all this time meaning it is falling behind, so the batches should be much more full than they actually are. ![Screen Shot 2020-05-07 at 10.06.32 AM](https://us1.discourse-cdn.com/elastic/original/3X/7/6/76f4654b61f028a0e54aaf4ddf2b612e934053bb.png) It seems to me like the bottleneck is between pulling the data from the persistent queue and putting it into the producer batches. I'm trying to understand how this could be the case though - the disk doesn't seem too busy and the Kafka cluster I'm pushing to is supporting other producers with much higher events per second than this example. ---
**Author:** ![taylorn](https://avatars.discourse-cdn.com/v4/letter/t/c6cbf5/32.png) [@taylorn](https://discuss.elastic.co/u/taylorn)\ **Post date:** [May 13, 2020, 10:20pm UTC](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426/4 "2020-05-13T22:20:37Z")
I got massively improved performance by setting pipeline.batch.size to a much higher value like 10000-20000 as well as turning off the persistent queue. I also set pipeline.batch.delay to 10000 to force larger batches. With persistent queue enabled the performance is about 80% worse, even though iostat made it seem like the disk wasn't fully utilized. The value for pipeline.batch.size seems very high high but so far it is giving me the best throughput in the load tests. There is a warning at startup about too many maximum inflight events, but so far I don't have OOM errors on a 6GB JVM. From all this, it seems to me like the TCP listener was not able to keep up with the input load very well due to the smaller batch size, and the disk or however Logstash uses the persistent queue is also a limiting factor. I was using Logstash 6.8.3 so maybe a newer version would help. ---
**Author:** ![taylorn](https://avatars.discourse-cdn.com/v4/letter/t/c6cbf5/32.png) [@taylorn](https://discuss.elastic.co/u/taylorn)\ **Post date:** [May 14, 2020, 10:23pm UTC](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426/5 "2020-05-14T22:23:45Z")
Further updates... increasing pipeline.workers to a higher value has improved performance of the part of the pipeline between the persistent queue and the Kafka producer, resulting in larger and more frequent batches. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 11, 2020, 10:23pm UTC](https://discuss.elastic.co/t/kafka-output-increasing-batching-reduces-throughput/231426/6 "2020-06-11T22:23:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.