Filebeat.inputs journald: Multiline not working

Elastic Stack Beats
1

According to [Journalbeat] Still no multiline support after 3 years · Issue #27578 · elastic/beats · GitHub, because filebeat can now read the journal and it should support multiline. In the documentation Journald input | Filebeat Reference [7.16] | Elastic, there is unfortunately no explicit mention to multiline yet, so I have tried long to come out with a working filebeat configuration to correctly parse java multiline exceptions.

Tested on:

$ filebeat version
filebeat version 7.16.3 (amd64), libbeat 7.16.3 [d420ccdaf201e32a524632b5da729522e50257ae built 2022-01-07 00:36:57 +0000 UTC]

I've first tested that I have a working config for the console:

filebeat.debug-console.yml

filebeat.inputs:
- type: stdin

  enabled: true

  multiline:
    type: pattern
    pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
    negate: false
    match: after

output.console:
  enabled: true

That correctly parses an arbitrary stacktrace in my output:

stacktrace.example.txt

Interrupted after 30 s
java.lang.InterruptedException: sleep interrupted
	at java.base/java.lang.Thread.sleep(Native Method)
        at many more...(Native Method)
	at java.base/java.lang.Thread.run(Thread.java:829)
Next output ...

filebeat run using input stdin (edited for readability)

$ cat stacktrace.example.txt | filebeat -c filebeat.debug-console.yml
{"@timestamp":"2022-01-14T11:21:17.978Z","message":"Interrupted after 30 s","input":{"type":"stdin"}}
{"@timestamp":"2022-01-14T11:21:17.978Z",input":{"type":"stdin"},"message":"java.lang.InterruptedException: sleep interrupted\n\tat java.base/java.lang.Thread.sleep(Native Method)\n\tat many more...(Native Method)\n\tat java.base/java.lang.Thread.run(Thread.java:829)"}
{"@timestamp":"2022-01-14T11:21:17.978Z","message":"Next output ...","input":{"type":"stdin"}}

Now, based on the previous configuration filebeat.debug-console.yml, I have created a new one for the filebeat.input type journald:

filebeat.debug-journald.yml

filebeat.inputs:
- type: journald

  enabled: true
  include_matches:
    - SYSLOG_IDENTIFIER=XXX

  multiline:
    type: pattern
    pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
    negate: false
    match: after

# This parsers section doesn't work either:
#  parsers:
#    - multiline:
#        type: pattern
#        pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
#        negate: false
#        match: after

output.console:
  enabled: true

But I couldn't get the multiline working:

filebeat run using input journald (edited for readability)

$ filebeat -c filebeat.debug-journald.yml & cat stacktrace.example.txt | systemd-cat -t XXX
[2] 47924
$ {"@timestamp":"2022-01-14T14:12:40.114Z","message":"Interrupted after 30 s","input":{"type":"journald"}}
{"@timestamp":"2022-01-14T14:12:40.114Z","input":{"type":"journald"},"message":"java.lang.InterruptedException: sleep interrupted"}
{"@timestamp":"2022-01-14T14:12:40.114Z","message":"\tat java.base/java.lang.Thread.sleep(Native Method)","input":{"type":"journald"}}
{"@timestamp":"2022-01-14T14:12:40.114Z","message":"\tat many more...(Native Method)","input":{"type":"journald"}}
{"@timestamp":"2022-01-14T14:12:40.114Z","message":"\tat java.base/java.lang.Thread.run(Thread.java:829)","input":{"type":"journald"}}
{"@timestamp":"2022-01-14T14:12:40.114Z","message":"Next output ...","input":{"type":"journald"}}

Notice that there are 6 events, instead of the expected 3.

2

Hello @agallardo

Based on the Filebeat journald input documentation I do not belive this input type supports multiline yet.

In the documentation it says:

The journald input supports the following configuration options plus the Common options described later.

Multiline support is not mentioned as common options.

Best regards
Flemming

3

Thanks @fgjensen . That is disappointing, as the github comment posted above suggests that it already supports multiline, being this one reason to deprecate journalbeat.
Additionally, the current documentation is confusing in this regard, as multiline has a dedicated section ( multiline) under "Inputs", that also suggest it is applicable to all input types.

I'll wait a bit for confirmation before I file a feature request in GitHub.

4

FR filed: [filebeat] Add multiline support for journald input · Issue #29907 · elastic/beats · GitHub

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.