Filebeat Multiline.Count_Lines

Elastic Stack Beats
1

Hello Community,

I'm currently using Filebeat to stream my logs to Kafka. Everything works as expected besides the multiline option. I'd like to bulk several lines into one event to stream bigger messages to Kafka as this is increasing my throughput to handle all incoming logs.

I tried with Filebeat 7.12.0 as well as 7.13.0.

Input: Log --> Output: Kafka

This is my config:

multiline.type: count
multiline.count_lines: 10
multiline.timeout: 3s

I was hoping that Filebeat aggregates 10 lines and if less lines are available, send the available lines after the timeout of 3s.
I have the problem that often at the end of my files not all logs are collected and transmitted when there are less then 10 lines left.

I hoping for any kind of help on this.
Best regards,
Beater

2

I discovered an interesting bug now as well:
After the close_inactive = 5m timeout is reached, instead of sending the pending lines, Filebeat will send the last 10 lines again.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.