Filebeat logrotation multiline data loss

[PeterPloug (Peter)]

Hi All
OS: red hat
filebeat version: 6.4.2
First of I am very new to the entire ELK stack and I am trying to replace Heka with filebeat/logstash.
I have an issue where log rotation is dividing a xml event across log files.
This means the event is partly in one log file and partly in another.
I have setup filebeat to read multiline using a pattern, which matches the specific xml structure. The problem is that I am loosing data when the xml event is big and the log file is rotated.
The multiline pattern works fine as long as the entire event is inside ONE log file.

I understand that filebeat is using multiple harvesters to read all new log files and what I am looking for is most likely a sequential reading of log files.

I need to parse the xml event in logstash or directly in filebeat (not decided yet).

Is there anyway for filebeat to handle this situation?

[pierhugues (Pier-Hugues Pellerin)]

@PeterPloug I think it's a limitation of Filebeat there, we effectively assume that a multiline log event is not splitted between different files. Supporting this would require us to change a few things and be able to undestand file order to make sure events are correctly merged together.

Filebeat doesn't currently support reading XML content, so I think the best approach is to use Logstash and the XML filter in that case.