Logstash to handle multiline input from Filebeat

Elastic Stack Logstash
1

Hi,

Would just like to ask how can Logstash handle multiline from input Filebeat.
My filebeat already handles the multiline however the logstash still outputs only the first line. after the \n all the rest are ignored already.
I tried to put the below filter but still not working.

mutate { 
        gsub => [ "message", "\\n", "" ] 
}

Please please advise. Thank you!

2

Please provide more detail of both the logstash and filebeat configurations, and if possible, a reproducible example. If I configure filebeat with

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/foo*.txt
  multiline.pattern: ^Monsieur Spalanzani
  multiline.negate: true
  multiline.match: after
  multiline.timeout: 5s
output.logstash:
  hosts: ["127.10.11.12:5044"]

and logstash with

input { beats { host => "127.10.11.12" port => 5044 } }

and append

Hoffmann
Monsieur Spalanzani n'aime pas la musique.
That's what she said!
Monsieur Spalanzani n'aime pas la musique.
Oui, je sais! oui, je sais:
tout pour la physique!

to /tmp/foo.txt then I get this set of messages

   "message" => "Hoffmann",
   "message" => "Monsieur Spalanzani n'aime pas la musique.\nThat's what she said!",
   "message" => "Monsieur Spalanzani n'aime pas la musique.\nOui, je sais! oui, je sais:\ntout pour la physique!",

In the gsub you do not need to escape the backslash. The second array entry (the "needle") is a processed as a regexp. So

mutate { gsub => [ "message", "\n", "" ] }

will change that last message to

"message" => "Monsieur Spalanzani n'aime pas la musique.Oui, je sais! oui, je sais:tout pour la physique!",
3

Hi Badger, thanks for your response.

Here's my Filebeat config regarding multiline

FILEBEAT:

filebeat.inputs:
- type: log
enabled: true
paths:
    - D:\Test\listen.log
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
output.logstash:
 hosts: ["collectornode.test.com:9513"]

LOGSTASH INPUT:

input {
	beats {
		id => "log-app-fab-input"
		port => 9513
        codec => plain
	}
}

SAMPLE FILEBEAT INPUT:

2021-11-25 03:28:26,787 [4] INFO  - subject=Test.Subject, reply=_INBOX.0A8FB204, message={HEADER={HOST="TESTSA1290" DOMAIN="EU" USER="TESTUser" LIB_VERSION="5.3.2 (8.4)" PID="6448" APP_NAME="Test.Application" APP_VERSION="1.0.0.0" REQ_MODE=true TIMESTAMP=2021-11-25 02:28:27.074998000Z MSG_ID="145450140157002" REQ_ID="" QoS="R"} DATA={$SERVICE="Request" Type="Query" ConnectionType="Sync" Data="<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<EDL xmlns="http://www.infineon.com/edl">
  <EDLHeader>
    <Output Type="EFF" />
    <Tags>
      <Item Source="eSquare" Name="IsAeSquareEDL">
        <Value>True</Value>
      </Item>
      <Item Source="eSquare" Name="E2XMLversion">
        <Value>2.1.0.0</Value>
      </Item>
      <Item Source="eSquare" Name="E2Clientversion">
        <Value>2.8.5.4</Value>
      </Item>
      <Item Source="eSquare" Name="E2Clientname">
        <Value>eSquare Main Application</Value>
      </Item>

FILEBEAT OUTPUT:

  "message": "2021-11-25 04:52:35,979 [4] INFO  - subject=Test.Subject, reply=_INBOX.0A8FB204, message={HEADER={HOST=\"TESTSA1290\" DOMAIN=\"EU\" USER=\"RBGE2\" LIB_VERSION=\"5.3.2 (8.4)\" PID=\"6448\" APP_NAME=\"TestApp\" APP_VERSION=\"1.0.0.0\" REQ_MODE=true TIMESTAMP=2021-11-25 03:52:34.844914000Z MSG_ID=\"126242838258952\" REQ_ID=\"\" QoS=\"R\"} DATA={$SERVICE=\"Request\" Type=\"Query\" ConnectionType=\"Sync\" Data=\"<?xml version=\"1.0\" encoding=\"iso-8859-1\" standalone=\"yes\"?>\n<EDL xmlns=\"http://www.test.com/edl\">\n  <EDLHeader>\n    <Output Type=\"EFF\" />\n    <Tags>\n      <Item Source=\"eSquare\" Name=\"IsAeSquareEDL\">\n        <Value>True</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"E2XMLversion\">\n        <Value>2.1.0.0</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"E2Clientversion\">\n        <Value>2.8.5.4</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"E2Clientname\">\n        <Value>eSquare Main Application</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"E2Repositoryversion\">\n        <Value>2.1.0.0</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"ExecutionID\">\n        <Value>2931c06c-0db9-4262-aefb-1542167cc04a</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"OwningUser\">\n        <Value>test\\bolender</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"JobDefinitionID\">\n        <Value>1003487346</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"JobDefinitionName\">\n        <Value>ARR-Lot_Disposition_grouping</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"LastModifiedUserLw\">\n        <Value>bolender_infineon</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"LastModifiedTimestampLw\">\n        <Value>10/7/2020 10:59:56 AM</Value>\n      </Item>\n      <Item Source=\"eSquare\" Name=\"LogMessage\">\n        <Value>This job definition 'ARR-Lot_Disposition_grouping' was exported</Value>\n      </Item>\n    </Tags>\n  </EDLHeader>\n  <DataQueryDefinition>\n ,

LOGSTASH OUTPUT:

	{HEADER={HOST="TESTSA1290" DOMAIN="EU" USER="RBGE2" LIB_VERSION="5.3.2 (8.4)" PID="6448" APP_NAME="TestApp" APP_VERSION="1.0.0.0" REQ_MODE=true TIMESTAMP=2021-11-25 02:28:27.074998000Z MSG_ID="145450140157002" REQ_ID="" QoS="R"} DATA={$SERVICE="Request" Type="Query" ConnectionType="Sync" Data="<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
4

That looks like your logstash configuration has overwritten the original [message] field with the value of the message={HEADER=... from inside that field.

My first impression is that the filebeat part is working OK. How are you parsing the[message] field in the filter section of logstash?

5

Hi Badger,

It works now!

This filter worked. I can see the whole message={} in Kibana now.

mutate { gsub => [ "message", "\n", "" ] }

Thank you so much for your help!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.