OS: red hat
filebeat version: 6.4.2
First of I am very new to the entire ELK stack and I am trying to replace Heka with filebeat/logstash.
I have an issue where log rotation is dividing a xml event across log files.
This means the event is partly in one log file and partly in another.
I have setup filebeat to read multiline using a pattern, which matches the specific xml structure. The problem is that I am loosing data when the xml event is big and the log file is rotated.
The multiline pattern works fine as long as the entire event is inside ONE log file.
I understand that filebeat is using multiple harvesters to read all new log files and what I am looking for is most likely a sequential reading of log files.
I need to parse the xml event in logstash or directly in filebeat (not decided yet).
Is there anyway for filebeat to handle this situation?