# Logstash to handle multiline input from Filebeat

**URL:** <https://discuss.elastic.co/t/logstash-to-handle-multiline-input-from-filebeat/290116>\
**Category:** Logstash\
**Created:** [November 25, 2021, 2:23am UTC](https://discuss.elastic.co/t/logstash-to-handle-multiline-input-from-filebeat/290116 "2021-11-25T02:23:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![grazia0912](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@grazia0912](https://discuss.elastic.co/u/grazia0912)\
**Post date:** [November 25, 2021, 2:23am UTC](https://discuss.elastic.co/t/logstash-to-handle-multiline-input-from-filebeat/290116/1 "2021-11-25T02:23:53Z")

</div>

Hi,

Would just like to ask how can Logstash handle multiline from input Filebeat.  
My filebeat already handles the multiline however the logstash still outputs only the first line. after the \n all the rest are ignored already.  
I tried to put the below filter but still not working.

```auto
mutate { 
        gsub => ["message", "\\n", ""] 
}

```

Please please advise. Thank you!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 25, 2021, 3:48am UTC](https://discuss.elastic.co/t/logstash-to-handle-multiline-input-from-filebeat/290116/2 "2021-11-25T03:48:26Z")

</div>

Please provide more detail of both the logstash and filebeat configurations, and if possible, a reproducible example. If I configure filebeat with

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/foo*.txt
  multiline.pattern: ^Monsieur Spalanzani
  multiline.negate: true
  multiline.match: after
  multiline.timeout: 5s
output.logstash:
  hosts: ["127.10.11.12:5044"]

```

and logstash with

```
input { beats { host => "127.10.11.12" port => 5044 } }

```

and append

```auto
Hoffmann
Monsieur Spalanzani n'aime pas la musique.
That's what she said!
Monsieur Spalanzani n'aime pas la musique.
Oui, je sais! oui, je sais:
tout pour la physique!

```

to /tmp/foo.txt then I get this set of messages

```
   "message" => "Hoffmann",
   "message" => "Monsieur Spalanzani n'aime pas la musique.\nThat's what she said!",
   "message" => "Monsieur Spalanzani n'aime pas la musique.\nOui, je sais! oui, je sais:\ntout pour la physique!",

```

In the gsub you do not need to escape the backslash. The second array entry (the "needle") is a processed as a regexp. So

```
mutate { gsub => ["message", "\n", ""] }

```

will change that last message to

```
"message" => "Monsieur Spalanzani n'aime pas la musique.Oui, je sais! oui, je sais:tout pour la physique!",
```

---

<div class="post-metadata">

**Author:** ![grazia0912](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@grazia0912](https://discuss.elastic.co/u/grazia0912)\
**Post date:** [November 25, 2021, 4:07am UTC](https://discuss.elastic.co/t/logstash-to-handle-multiline-input-from-filebeat/290116/3 "2021-11-25T04:07:58Z")

</div>

Hi Badger, thanks for your response.

Here's my Filebeat config regarding multiline

FILEBEAT:

```auto
filebeat.inputs:
- type: log
enabled: true
paths:
    - D:\Test\listen.log
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
output.logstash:
 hosts: ["collectornode.test.com:9513"]

```

LOGSTASH INPUT:

```auto
input {
	beats {
		id => "log-app-fab-input"
		port => 9513
        codec => plain
	}
}

```

SAMPLE FILEBEAT INPUT:

```auto
2021-11-25 03:28:26,787 [4] INFO - subject=Test.Subject, reply=_INBOX.0A8FB204, message={HEADER={HOST="TESTSA1290" DOMAIN="EU" USER="TESTUser" LIB_VERSION="5.3.2 (8.4)" PID="6448" APP_NAME="Test.Application" APP_VERSION="1.0.0.0" REQ_MODE=true TIMESTAMP=2021-11-25 02:28:27.074998000Z MSG_ID="145450140157002" REQ_ID="" QoS="R"} DATA={$SERVICE="Request" Type="Query" ConnectionType="Sync" Data="<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<EDL xmlns="http://www.infineon.com/edl">
  <EDLHeader>
    <Output Type="EFF" />
    <Tags>
      <Item Source="eSquare" Name="IsAeSquareEDL">
        <Value>True</Value>
      </Item>
      <Item Source="eSquare" Name="E2XMLversion">
        <Value>2.1.0.0</Value>
      </Item>
      <Item Source="eSquare" Name="E2Clientversion">
        <Value>2.8.5.4</Value>
      </Item>
      <Item Source="eSquare" Name="E2Clientname">
        <Value>eSquare Main Application</Value>
      </Item>

```

FILEBEAT OUTPUT:

```auto
  "message": "2021-11-25 04:52:35,979 [4] INFO - subject=Test.Subject, reply=_INBOX.0A8FB204, message={HEADER={HOST=\"TESTSA1290\" DOMAIN=\"EU\" USER=\"RBGE2\" LIB_VERSION=\"5.3.2 (8.4)\" PID=\"6448\" APP_NAME=\"TestApp\" APP_VERSION=\"1.0.0.0\" REQ_MODE=true TIMESTAMP=2021-11-25 03:52:34.844914000Z MSG_ID=\"126242838258952\" REQ_ID=\"\" QoS=\"R\"} DATA={$SERVICE=\"Request\" Type=\"Query\" ConnectionType=\"Sync\" Data=\"<?xml version=\"1.0\" encoding=\"iso-8859-1\" standalone=\"yes\"?>\n<EDL xmlns=\"http://www.test.com/edl\">\n <EDLHeader>\n <Output Type=\"EFF\" />\n <Tags>\n <Item Source=\"eSquare\" Name=\"IsAeSquareEDL\">\n <Value>True</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"E2XMLversion\">\n <Value>2.1.0.0</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"E2Clientversion\">\n <Value>2.8.5.4</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"E2Clientname\">\n <Value>eSquare Main Application</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"E2Repositoryversion\">\n <Value>2.1.0.0</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"ExecutionID\">\n <Value>2931c06c-0db9-4262-aefb-1542167cc04a</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"OwningUser\">\n <Value>test\\bolender</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"JobDefinitionID\">\n <Value>1003487346</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"JobDefinitionName\">\n <Value>ARR-Lot_Disposition_grouping</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"LastModifiedUserLw\">\n <Value>bolender_infineon</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"LastModifiedTimestampLw\">\n <Value>10/7/2020 10:59:56 AM</Value>\n </Item>\n <Item Source=\"eSquare\" Name=\"LogMessage\">\n <Value>This job definition 'ARR-Lot_Disposition_grouping' was exported</Value>\n </Item>\n </Tags>\n </EDLHeader>\n <DataQueryDefinition>\n ,

```

LOGSTASH OUTPUT:

```auto
	{HEADER={HOST="TESTSA1290" DOMAIN="EU" USER="RBGE2" LIB_VERSION="5.3.2 (8.4)" PID="6448" APP_NAME="TestApp" APP_VERSION="1.0.0.0" REQ_MODE=true TIMESTAMP=2021-11-25 02:28:27.074998000Z MSG_ID="145450140157002" REQ_ID="" QoS="R"} DATA={$SERVICE="Request" Type="Query" ConnectionType="Sync" Data="<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 25, 2021, 4:26am UTC](https://discuss.elastic.co/t/logstash-to-handle-multiline-input-from-filebeat/290116/4 "2021-11-25T04:26:48Z")

</div>

That looks like your logstash configuration has overwritten the original [message] field with the value of the `message={HEADER=...` from inside that field.

My first impression is that the filebeat part is working OK. How are you parsing the[message] field in the filter section of logstash?

---

<div class="post-metadata">

**Author:** ![grazia0912](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@grazia0912](https://discuss.elastic.co/u/grazia0912)\
**Post date:** [November 25, 2021, 5:25am UTC](https://discuss.elastic.co/t/logstash-to-handle-multiline-input-from-filebeat/290116/5 "2021-11-25T05:25:28Z")

</div>

Hi Badger,

It works now!

This filter worked. I can see the whole message={} in Kibana now.

```auto
mutate { gsub => ["message", "\n", ""] }

```

Thank you so much for your help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 23, 2021, 5:25am UTC](https://discuss.elastic.co/t/logstash-to-handle-multiline-input-from-filebeat/290116/6 "2021-12-23T05:25:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.