# Performance Guide Needed

**URL:** <https://discuss.elastic.co/t/performance-guide-needed/175867>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 8, 2019, 3:09pm UTC](https://discuss.elastic.co/t/performance-guide-needed/175867 "2019-04-08T15:09:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Or\_Arnon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/or_arnon/32/43677_2.png) [@Or\_Arnon](https://discuss.elastic.co/u/Or_Arnon)\
**Post date:** [April 8, 2019, 3:09pm UTC](https://discuss.elastic.co/t/performance-guide-needed/175867/1 "2019-04-08T15:09:54Z")

</div>

Hi,

We're in the process of optimizing our Filebeat processes and we're looking for a general guideline of how to do so.  
We have multiple HAproxy servers which write logs via rsyslog. Each event is written in JSON and its size is 1.5K to 2.0K. File can have between 6M - 10M events.  
We are sending these events to Kafka.

Currently, we set our Filebeat to 4 workers with a bulk\_max\_size of 4098.

We're looking for ways to benchmark and/or best practices for Kafka.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 9, 2019, 1:04am UTC](https://discuss.elastic.co/t/performance-guide-needed/175867/2 "2019-04-09T01:04:25Z")

</div>

What problems are you looking to solve with this optimisation?

---

<div class="post-metadata">

**Author:** ![Or\_Arnon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/or_arnon/32/43677_2.png) [@Or\_Arnon](https://discuss.elastic.co/u/Or_Arnon)\
**Post date:** [April 10, 2019, 8:32am UTC](https://discuss.elastic.co/t/performance-guide-needed/175867/3 "2019-04-10T08:32:31Z")

</div>

We're pretty sure we can increase the consuming rate of Filebeat from these files and send them to Kafka. Batch size, number of workers and other parameters can be adjusted. We're looking for a way to go about it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2019, 8:33am UTC](https://discuss.elastic.co/t/performance-guide-needed/175867/4 "2019-05-08T08:33:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.