Filebeat logrotation multiline data loss

Hi All
OS: red hat
filebeat version: 6.4.2
First of I am very new to the entire ELK stack and I am trying to replace Heka with filebeat/logstash.
I have an issue where log rotation is dividing a xml event across log files.
This means the event is partly in one log file and partly in another.
I have setup filebeat to read multiline using a pattern, which matches the specific xml structure. The problem is that I am loosing data when the xml event is big and the log file is rotated.
The multiline pattern works fine as long as the entire event is inside ONE log file.

I understand that filebeat is using multiple harvesters to read all new log files and what I am looking for is most likely a sequential reading of log files.

I need to parse the xml event in logstash or directly in filebeat (not decided yet).

Is there anyway for filebeat to handle this situation?

@PeterPloug I think it's a limitation of Filebeat there, we effectively assume that a multiline log event is not splitted between different files. Supporting this would require us to change a few things and be able to undestand file order to make sure events are correctly merged together.

Filebeat doesn't currently support reading XML content, so I think the best approach is to use Logstash and the XML filter in that case.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.