Filebeat reads logfiles, processes multiline and sends to logstesh. (all OK)
But at the time of file rotation, the filebeat cuts the message into two parts, the first in the rotated file and the second in the new file. I could not get around this problem.
Using the pattern, I can define the first message and the second. I used the path to the file as an identifier.
I can not use multiline logstesh because many sources. Using one worker is also not acceptable, so you have to define the first and second message with patterns, insert them into an array, and then convert them to a string.
I think that at the moment the filebeat stops, it is likely that the multilines will not stop at the end of the message. To them we get again two parts.
At the moment when the rotation occurs, the file is closed and the multiline does not reach the last line of the message.
The new log file contains lines of the second half of the message, but the filebeat combines them as a new message.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.