Filebeat exclude lines with multiline

I can only seem to get exlude_lines to work, if multiline is not enabled. I suspect that this due to the order in which these directives are processed. My guess is that the multiline is processed first, which would then make the exclude_lines not have a match to work with. Can anyone confirm that, or show me how my config is wrong?


- type: log
  enabled: true
     - /home/eti/logtest/triad-current-msg-format.log
        log_type: triad-current-msg
  # Exlude the line of dashes TODO this seems to get ignored when multiline is working. Probably need to strip it in logstash
  exclude_lines: ['^-+$']
  # Setup the pattern to harvest the multiline
  multiline.pattern: '^[A-Z]+: '
  multiline.negate: true
  multiline.match: after

Log Pattern

ERROR: 12/19/18 02:16:00.225 PID=3126 (cbppvd 1000)
Database Error: Function=add_package Stmt=insert ppvpacks in cborg2001, pack_event_nbr=198740 event_nbr=234824
Code -691: Missing key in referenced table for referential constraint (root.r211_1274).
ISAM Code -111: ISAM error:  no record found.


Hi @swright-eti,

Yes, as you suppose the multiline is processed first, so the problem is that the line with the dashes is considered part of the previous multiline, so it doesn't match. I guess that in your example you see that the last two lines are sent in the same event:

ISAM Code -111: ISAM error:  no record found.

If your logs always start with [A-Z]+: one thing you can try is to add a pattern to the multiline so a line with dashes is considered its own multiline event, then it should be excluded by exclude_lines. Something like this:

multiline.pattern: '(^[A-Z]+: |^-+$)'

Thanks. I ended up using mutate in Logstash config.

mutate {
    gsub => [
        # Replace the line of dashes
        "message", "-+$", ""

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.