Aggregate. Merging two messages in different order

Elastic Stack Logstash
1

Hello everyone!

Filebeat 7.6.1 -> logstash 7.6.1

Filebeat reads logfiles, processes multiline and sends to logstesh. (all OK)
But at the time of file rotation, the filebeat cuts the message into two parts, the first in the rotated file and the second in the new file. I could not get around this problem.

For the solution I want to use the filter Aggregate on the Logstesh.
But I don't know much about Ruby and have never used this filter.
I searched the Internet for an example: https://stackoverflow.com/questions/51421183/filebeat-logstash-multiline-syslog-parsing

Using the pattern, I can define the first message and the second. I used the path to the file as an identifier.
I can not use multiline logstesh because many sources. Using one worker is also not acceptable, so you have to define the first and second message with patterns, insert them into an array, and then convert them to a string.

if [message] =~ /pattern_first_message/ {
	aggregate {
		task_id => "%{file_path}"
		code => "map['message'] ||= []; map['message'].unshift(event.get('message'));"
		push_map_as_event_on_timeout => true
		timeout => 10
		timeout_tags => "_aggregate"
		timeout_code => "event.set('message', map['message'].join(' '))"
	}
} else if [message] =~ /pattern_second_message/ {
	aggregate {
		task_id => "%{file_path}"
		code => "map['message'] ||= []; map['message'].push(event.get('message'));"
	}
}

When processing such a filter, I get an error:
Aggregate exception occurred {:error=>#<NameError: undefined local variable or method map' for`

Could you help fix my filter?

2

Hi,

Are you using loggrotate from linux ?

If yes a simple fix would be to temporarly stop filebeat during the rotation ( script ) then start-it right after, you should be fine.

3

I think that at the moment the filebeat stops, it is likely that the multilines will not stop at the end of the message. To them we get again two parts.

4

Hi,

I dont think i understand your problem, can you be more specific when you say

But at the time of file rotation, the filebeat cuts the message into two parts

What rotation ?

Can you send me your filebeat configuration ?

The problem you are trying to fix with logstash should not exist in the first place.

5

Files that filebeat read are constantly being filled with a syslog server. These files are configured with the rotation utility logrotate.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/patternfilename*.log

  line_terminator: line_feed
  scan_frequency: 1s

  multiline.pattern: 'stringpattern'
  multiline.negate: true
  multiline.match: after
  multiline.timeout: 5s

  close_renamed: false
  close_removed: true

At the moment when the rotation occurs, the file is closed and the multiline does not reach the last line of the message.
The new log file contains lines of the second half of the message, but the filebeat combines them as a new message.