# How to slow down large amount of data coming from filebeat?

**URL:** <https://discuss.elastic.co/t/how-to-slow-down-large-amount-of-data-coming-from-filebeat/224970>\
**Category:** Logstash\
**Created:** [March 25, 2020, 10:09am UTC](https://discuss.elastic.co/t/how-to-slow-down-large-amount-of-data-coming-from-filebeat/224970 "2020-03-25T10:09:51Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 26, 2020, 9:37am UTC](https://discuss.elastic.co/t/how-to-slow-down-large-amount-of-data-coming-from-filebeat/224970/6 "2020-03-26T09:37:25Z")

</div>

@A_B as you make the move to Kafka, a few things that will really boost throughput...

1. increase `pipeline.batch.size` from the default of 125 to at least 1024 (1280 was best in my environment)
2. increase `pipeline.batch.delay` from the default of 50 to at least 500 (1000 was best in my environment)
3. in the `kafka` input, set `max_poll_records` to the same value as `pipeline.batch.size`
4. each thread defined by `consumer_threads` in the `kafka` input will be an instance of a consumer. So if you have 4 instances with 2 threads, that is 8 consumer instances. Your Kafka topics must have at least 8 partitions for all consumer threads to ingest data. You will want more partitions than your current needs so you can easily scale in the future.
5. the number of `pipeline.workers` should be at least equal to `consumer_threads`.
6. the kafka output should set `batch_size` to at least 16384

You may end up tweaking some of the buffer settings as well, but the above will give you a good starting point.

Rob

[![GitHub](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f8ae834f16b1a02d31607317669716807844d84.png)](https://github.com/robcowart) [![YouTube](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43b9b81a8c93786219985aeb5335c1c323449053.png)](https://www.youtube.com/channel/UCivWvTx1DwrWNcDLV58kmOg) [![LinkedIn](https://us1.discourse-cdn.com/elastic/original/3X/6/7/674f3370d0f0542ddc5e408516beb1b7edd6c1bf.png)](https://www.linkedin.com/in/robertcowart/)  
**[How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations](https://www.youtube.com/watch?v=gZb7HpVOges)**  
**[What is the best storage technology for Elasticsearch?](https://www.youtube.com/watch?v=nKUpfJCBiS4)**

---

_[View the full topic](https://discuss.elastic.co/t/how-to-slow-down-large-amount-of-data-coming-from-filebeat/224970)._