I'm currently using Filebeat to stream my logs to Kafka. Everything works as expected besides the multiline option. I'd like to bulk several lines into one event to stream bigger messages to Kafka as this is increasing my throughput to handle all incoming logs.
I tried with Filebeat 7.12.0 as well as 7.13.0.
Input: Log --> Output: Kafka
This is my config:
multiline.type: count multiline.count_lines: 10 multiline.timeout: 3s
I was hoping that Filebeat aggregates 10 lines and if less lines are available, send the available lines after the timeout of 3s.
I have the problem that often at the end of my files not all logs are collected and transmitted when there are less then 10 lines left.
I hoping for any kind of help on this.