Performance Guide Needed

Elastic Stack Beats
1

Hi,

We're in the process of optimizing our Filebeat processes and we're looking for a general guideline of how to do so.
We have multiple HAproxy servers which write logs via rsyslog. Each event is written in JSON and its size is 1.5K to 2.0K. File can have between 6M - 10M events.
We are sending these events to Kafka.

Currently, we set our Filebeat to 4 workers with a bulk_max_size of 4098.

We're looking for ways to benchmark and/or best practices for Kafka.

2

What problems are you looking to solve with this optimisation?

3

We're pretty sure we can increase the consuming rate of Filebeat from these files and send them to Kafka. Batch size, number of workers and other parameters can be adjusted. We're looking for a way to go about it.