Filebeat multiline ignores last line

Elastic Stack Beats
1

What I want to do is read these records, each of them is inside braces, so I use multilines in filebeat to be able to read them together, however, the last line "]}" is not read by filebeat, so the record is unfinished and the grok configuration of logstash fails

This is how they enter the data

{C-FLOW-ID-CAB APN101MQ C-OPERATION-CAB P T-EVENTO-CAB RUNNING T-EXTERNAL-ID-CAB NULL F-MESSAGE-CAB 20221110 H-MESSAGE-CAB 600478 C-MESSAGE-ID-CAB DC62CCED0E5F6000 M-STATUS-CAB 02 [
<FOTO><Status>RUNNING</Status><EVENTO>2022111006:00:47.3</EVENTO></FOTO>
]}
{C-FLOW-ID-CAB APN101MQ C-OPERATION-CAB P T-EVENTO-CAB RUNNING T-EXTERNAL-ID-CAB NULL F-MESSAGE-CAB 20221110 H-MESSAGE-CAB 1400584 C-MESSAGE-ID-CAB DC633840E86BB000 M-STATUS-CAB 02 [
<FOTO><Status>RUNNING</Status><EVENTO>2022111014:00:57.8</EVENTO></FOTO>
]}

In this way it is expected that the message arrives (with the "]}" at the end)

image
image1586×35 2.96 KB

But the last records arrives this way (without the "]}", which is the last line)

image
image1586×40 3.11 KB

this is my multiline configuration

 parsers:
  - multiline:
     type: pattern
     pattern: '^{'
     negate: true
     match: after  
     skip_newline: true

grok configuration

grok {		
		match => { 'message' => '^{(?:[^:]+) %{WORD:C-FLOW-ID-CAB} (?:[^:]+) %{WORD:C-OPERATION-CAB} (?:[^:]+) %{WORD:T-EVENTO-CAB} (?:[^:]+) %{WORD:T-EXTERNAL-ID-CAB} (?:[^:]+) %{WORD:F-MESSAGE-CAB} (?:[^:]+) %{WORD:H-MESSAGE-CAB} (?:[^:]+) %{WORD:C-MESSAGE-ID-CAB} (?:[^:]+) %{WORD:M-STATUS-CAB} \[%{GREEDYDATA:DETAIL}\]\}'}
	}
2

Hi @mariana17

Filebeat detects the end of an event by way of a new line character.
When the file receives the final ]} string, do you know if it has a new line character afterward?
If not, then filebeat will not process it as an event in order to append it to the previous multi-line event.

3

so, is it recommended to place the start pattern at the end of the file so that it takes the last record?

4

I think regardless of your pattern, in order for an event to be collected by filebeat, the line in the log will have to end with a new-line character.

Are these application logs that can be modified to allow for a new-line after the final ]}?

Do the logs lend themselves to allow you to end with </FOTO> instead?
If so, you can always drop or exclude the final ]}.

Another thought could be to use the multiline.max_lines or multiline.count_lines and set it to something like 2. Again, this all depends on the variability of these logs.