# Filebeat multiline ignores last line

**URL:** <https://discuss.elastic.co/t/filebeat-multiline-ignores-last-line/325654>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 15, 2023, 6:50pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-last-line/325654 "2023-02-15T18:50:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mariana17](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@mariana17](https://discuss.elastic.co/u/mariana17)\
**Post date:** [February 15, 2023, 6:50pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-last-line/325654/1 "2023-02-15T18:50:50Z")

</div>

What I want to do is read these records, each of them is inside braces, so I use multilines in filebeat to be able to read them together, however, the last line "]}" is not read by filebeat, so the record is unfinished and the grok configuration of logstash fails

This is how they enter the data

```auto
{C-FLOW-ID-CAB APN101MQ C-OPERATION-CAB P T-EVENTO-CAB RUNNING T-EXTERNAL-ID-CAB NULL F-MESSAGE-CAB 20221110 H-MESSAGE-CAB 600478 C-MESSAGE-ID-CAB DC62CCED0E5F6000 M-STATUS-CAB 02 [
<FOTO><Status>RUNNING</Status><EVENTO>2022111006:00:47.3</EVENTO></FOTO>
]}
{C-FLOW-ID-CAB APN101MQ C-OPERATION-CAB P T-EVENTO-CAB RUNNING T-EXTERNAL-ID-CAB NULL F-MESSAGE-CAB 20221110 H-MESSAGE-CAB 1400584 C-MESSAGE-ID-CAB DC633840E86BB000 M-STATUS-CAB 02 [
<FOTO><Status>RUNNING</Status><EVENTO>2022111014:00:57.8</EVENTO></FOTO>
]}

```

In this way it is expected that the message arrives (with the "]}" at the end)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/f/1fa5f24cba4047c38483f754630a9e09bb0eae9c.png)

But the last records arrives this way (without the "]}", which is the last line)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66d70b8988bccff6f4fe67672bbbd4367ad86161.png)

this is my multiline configuration

```auto
 parsers:
  - multiline:
     type: pattern
     pattern: '^{'
     negate: true
     match: after  
     skip_newline: true

```

grok configuration

```auto
grok {		
		match => { 'message' => '^{(?:[^:]+) %{WORD:C-FLOW-ID-CAB} (?:[^:]+) %{WORD:C-OPERATION-CAB} (?:[^:]+) %{WORD:T-EVENTO-CAB} (?:[^:]+) %{WORD:T-EXTERNAL-ID-CAB} (?:[^:]+) %{WORD:F-MESSAGE-CAB} (?:[^:]+) %{WORD:H-MESSAGE-CAB} (?:[^:]+) %{WORD:C-MESSAGE-ID-CAB} (?:[^:]+) %{WORD:M-STATUS-CAB} \[%{GREEDYDATA:DETAIL}\]\}'}
	}

```

---

<div class="post-metadata">

**Author:** ![eMitch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emitch/32/93607_2.png) [@eMitch](https://discuss.elastic.co/u/eMitch)\
**Post date:** [February 15, 2023, 7:50pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-last-line/325654/2 "2023-02-15T19:50:19Z")

</div>

Hi @mariana17

Filebeat detects the end of an event by way of a new line character.  
When the file receives the final `]}` string, do you know if it has a new line character afterward?  
If not, then filebeat will not process it as an event in order to append it to the previous multi-line event.

---

<div class="post-metadata">

**Author:** ![mariana17](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@mariana17](https://discuss.elastic.co/u/mariana17)\
**Post date:** [February 15, 2023, 8:16pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-last-line/325654/3 "2023-02-15T20:16:18Z")

</div>

so, is it recommended to place the start pattern at the end of the file so that it takes the last record?

---

<div class="post-metadata">

**Author:** ![eMitch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emitch/32/93607_2.png) [@eMitch](https://discuss.elastic.co/u/eMitch)\
**Post date:** [February 15, 2023, 9:47pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-last-line/325654/4 "2023-02-15T21:47:40Z")

</div>

I think regardless of your pattern, in order for an event to be collected by filebeat, the line in the log will have to end with a new-line character.

Are these application logs that can be modified to allow for a new-line after the final `]}`?

Do the logs lend themselves to allow you to end with `</FOTO>` instead?  
If so, you can always drop or exclude the final `]}`.

Another thought could be to use the `multiline.max_lines` or `multiline.count_lines` and set it to something like 2. Again, this all depends on the variability of these logs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 15, 2023, 11:47pm UTC](https://discuss.elastic.co/t/filebeat-multiline-ignores-last-line/325654/5 "2023-03-15T23:47:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.