Merge Multiple Line Event Messages in Elastic through Filebeat

Elastic Stack Beats
1

Hi All,

We have a situation where we are monitoring some server syslogs using Filebeat/Logstash and we want log messages in multiple lines to show up as a single event in Elastic instead of an event for each line.
I checked this link Manage multiline messages | Filebeat Reference [8.14] | Elastic and it does talk about handling the multiple line events by making a change in the filebeat.yml file. I tried this config settings with no luck. Can you please help me with the issue

Below is the example from the syslog. It has the SQL execution time and SQL ID in different lines and we want them to show up in one event in Elastic.
=========================================================
INFO - 2024/06/04-10:18:33.522 UTC - UTCLAP70Y23Z2.18098.01.poznantx.00481.Mux.Svr - ===>Took 2891.968 seconds to execute that SQL (returning 40328570 rows) - ABC.POM at /scratch/yytcadm/abc142072023080300_502239ln/src/foundation/pom/eim/log/ExplainPlan.cxx(302)
Execution Plan:
SQL_ID ABCDEFGHIJK, child number 0

================================================

And, Below is my filebeat.yml config :
# ============================== Filebeat inputs ===============================

filebeat.inputs:
parsers:
- multiline:

  • type: pattern*
  • pattern: '^INFO'*
  • negate: true*
  • match: after*

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input-specific configurations.

# filestream is an input for collecting log messages from files.

  • type: filestream
    ====================================================
2

Hi,

try with this:

filebeat.inputs:
- type: filestream
  paths:
    - /path/to/your/logs/*.log
  parsers:
    - multiline:
        type: pattern
        pattern: '^INFO'
        negate: true
        match: after

Regards

3

Thank you @yago82 this was very helpful. I just gave it a try and voila it started working like a charm.
Thanks a bunch for your input!!! Much Appreciated :slight_smile: