Filebeat multiline based on capture group

Elastic Stack Beats
1

I'm trying to group multiline messages belonging to the same event. Unfortunately they don't appear to match the style of the examples in multiline-examples.html#_examples_of_multiline_configuration.

Each event has a unique 9-character alphanumeric value that should be easily isolated, and I can't figure out how to do that.

The number of lines per event can be dynamic so isolating "New ApplicationName SOAP request" as the start and line counting won't work.

I've trouble a couple approaches, most recently:

    multiline.type: while_pattern
    multiline.pattern: '^.*[ApplicationName:(.*)].*$'
    multiline.negate: false
    multiline.match: after

Example of the logfile entry:

[2024-03-10 00:07:16] [127.0.0.1:35822] [ApplicationName:D6NIHEW3] New ApplicationName SOAP request
[2024-03-10 00:07:16] [127.0.0.1:35822] [ApplicationName:D6NIHEW3] > Username: some_user_name
[2024-03-10 00:07:16] [127.0.0.1:35822] [ApplicationName:D6NIHEW3] > Password: xxxxxxxxxxx
[2024-03-10 00:07:16] [127.0.0.1:35822] [ApplicationName:D6NIHEW3] > Client ID: some_ip_address
[2024-03-10 00:07:16] [127.0.0.1:35822] [ApplicationName:D6NIHEW3] > Source IP: some_ip_address
[2024-03-10 00:07:16] [127.0.0.1:35822] [ApplicationName:D6NIHEW3] Registered ApplicationName request
[2024-03-10 00:07:16] [127.0.0.1:35822] [ApplicationName:D6NIHEW3] User invalid or not found
[2024-03-10 00:07:16] [127.0.0.1:35822] [ApplicationName:D6NIHEW3] Sent failure response

Thank you!

2

Hi, what version of filebeat?
what type are you using? I had a similar problem, perhaps the problem is in the config. I finally got multiline working. To understand all the pictures, it would be nice to see a little more of the log in order to understand the beginning and end of the multiline

3

My settings/version are:

# filebeat version
filebeat version 8.6.2 (amd64), libbeat 8.6.2 [9b77c2c135c228c2eedc310f6e975bb1a76169b1 built 2023-02-12 04:37:19 +0000 UTC]

filebeat.yml:
filebeat.inputs:
  - type: filestream
    id: my_filebeat_id_for_this
    enabled: true
    paths:
      - /root/test_file.log
    multiline.type: while_pattern
    multiline.pattern: '^.*[ApplicationName:(.*)].*$'
    multiline.negate: false
    multiline.match: after

output.console:
  pretty: true
4

Hi.
I'm using following config

- type: filestream
  id: tomcat-catalina
  tags: ["tomcat"]
  enabled: true 
  ignore_older: 24h
  paths:
    - /var/log/tomcat/catalina.*.log
  parsers:
    - multiline:
       type: pattern
       pattern: '^\w{3}\s[0-9]{2}'
       negate: true
       match: after

I think there is also a question with the log format itself. You need to understand the structure of the logs so that you can choose the right configuration and pattern

5

I've truncated the log example and am still struggling to get the result I want. I get filebeat to match patterns, but it globs them all together into a single message.

Using this as a test log format:

[ApplicationName:D6NIHEW3] New ApplicationName SOAP request
[ApplicationName:D6NIHEW3] > Username: some_user_name
[ApplicationName:D6NIHEW3] > Password: xxxxxxxxxxx
[ApplicationName:D6NIHEW3] > Client ID: some_ip_address
[ApplicationName:D6NIHEW3] > Source IP: some_ip_address
[ApplicationName:D6NIHEW3] Registered ApplicationName request
[ApplicationName:D6NIHEW3] User invalid or not found
[ApplicationName:D6NIHEW3] Sent failure response
[ApplicationName:G6H4564S] New ApplicationName SOAP request
[ApplicationName:G6H4564S] > Username: some_other_user_name
[ApplicationName:G6H4564S] > Password: xxxxxxxxxxx
[ApplicationName:G6H4564S] > Client ID: some_ip_address
[ApplicationName:G6H4564S] > Source IP: some_ip_address
[ApplicationName:G6H4564S] Registered ApplicationName request
[ApplicationName:G6H4564S] User invalid or not found
[ApplicationName:G6H4564S] Sent failure response

Guess I'm wondering if I can have each while_pattern track the unique event ID's D6NIHEW3 & G6H4564S to log as different messages.

Latest test config:

filebeat.inputs:
  - type: filestream
    id: id_I_am_using
    enabled: true
    paths:
      - '/tmp/13'
    parsers:
      - multiline:
          type: while_pattern
          pattern: '^\[ApplicationName:[A-Z0-9]+\]'
          negate: false
          match: after
6

Actually, I may have something.. if I set the pattern to include "New openotpSimpleLogin SOAP request", and set negate: true, I get the result I want.

pattern: '^\[ApplicationName:[A-Z0-9]+\] New openotpSimpleLogin SOAP request$'

However, I feel there should be a way to just match a unique [[:alnum:]] in every message and treat it as a multiline.