Filebeat multiline ignores last line

What I want to do is read these records, each of them is inside braces, so I use multilines in filebeat to be able to read them together, however, the last line "]}" is not read by filebeat, so the record is unfinished and the grok configuration of logstash fails

This is how they enter the data

{C-FLOW-ID-CAB APN101MQ C-OPERATION-CAB P T-EVENTO-CAB RUNNING T-EXTERNAL-ID-CAB NULL F-MESSAGE-CAB 20221110 H-MESSAGE-CAB 600478 C-MESSAGE-ID-CAB DC62CCED0E5F6000 M-STATUS-CAB 02 [
<FOTO><Status>RUNNING</Status><EVENTO>2022111006:00:47.3</EVENTO></FOTO>
]}
{C-FLOW-ID-CAB APN101MQ C-OPERATION-CAB P T-EVENTO-CAB RUNNING T-EXTERNAL-ID-CAB NULL F-MESSAGE-CAB 20221110 H-MESSAGE-CAB 1400584 C-MESSAGE-ID-CAB DC633840E86BB000 M-STATUS-CAB 02 [
<FOTO><Status>RUNNING</Status><EVENTO>2022111014:00:57.8</EVENTO></FOTO>
]}

In this way it is expected that the message arrives (with the "]}" at the end)

But the last records arrives this way (without the "]}", which is the last line)

this is my multiline configuration

 parsers:
  - multiline:
     type: pattern
     pattern: '^{'
     negate: true
     match: after  
     skip_newline: true

grok configuration

grok {		
		match => { 'message' => '^{(?:[^:]+) %{WORD:C-FLOW-ID-CAB} (?:[^:]+) %{WORD:C-OPERATION-CAB} (?:[^:]+) %{WORD:T-EVENTO-CAB} (?:[^:]+) %{WORD:T-EXTERNAL-ID-CAB} (?:[^:]+) %{WORD:F-MESSAGE-CAB} (?:[^:]+) %{WORD:H-MESSAGE-CAB} (?:[^:]+) %{WORD:C-MESSAGE-ID-CAB} (?:[^:]+) %{WORD:M-STATUS-CAB} \[%{GREEDYDATA:DETAIL}\]\}'}
	}

Hi @mariana17

Filebeat detects the end of an event by way of a new line character.
When the file receives the final ]} string, do you know if it has a new line character afterward?
If not, then filebeat will not process it as an event in order to append it to the previous multi-line event.

so, is it recommended to place the start pattern at the end of the file so that it takes the last record?

I think regardless of your pattern, in order for an event to be collected by filebeat, the line in the log will have to end with a new-line character.

Are these application logs that can be modified to allow for a new-line after the final ]}?

Do the logs lend themselves to allow you to end with </FOTO> instead?
If so, you can always drop or exclude the final ]}.

Another thought could be to use the multiline.max_lines or multiline.count_lines and set it to something like 2. Again, this all depends on the variability of these logs.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.