Filebeat exclude lines with multiline

[swright-eti (Steve)]

I can only seem to get exlude_lines to work, if multiline is not enabled. I suspect that this due to the order in which these directives are processed. My guess is that the multiline is processed first, which would then make the exclude_lines not have a match to work with. Can anyone confirm that, or show me how my config is wrong?

Config

- type: log
  enabled: true
  paths:
     - /home/eti/logtest/triad-current-msg-format.log
  fields:
        log_type: triad-current-msg
  # Exlude the line of dashes TODO this seems to get ignored when multiline is working. Probably need to strip it in logstash
  exclude_lines: ['^-+$']
  # Setup the pattern to harvest the multiline
  multiline.pattern: '^[A-Z]+: '
  multiline.negate: true
  multiline.match: after

Log Pattern

ERROR: 12/19/18 02:16:00.225 PID=3126 (cbppvd 1000)
Database Error: Function=add_package Stmt=insert ppvpacks in cborg2001, pack_event_nbr=198740 event_nbr=234824
Code -691: Missing key in referenced table for referential constraint (root.r211_1274).
ISAM Code -111: ISAM error:  no record found.
------------------------------------------------------------------------------

Thanks.

[jsoriano (Jaime Soriano)]

Hi @swright-eti,

Yes, as you suppose the multiline is processed first, so the problem is that the line with the dashes is considered part of the previous multiline, so it doesn't match. I guess that in your example you see that the last two lines are sent in the same event:

ISAM Code -111: ISAM error:  no record found.
------------------------------------------------------------------------------

If your logs always start with [A-Z]+: one thing you can try is to add a pattern to the multiline so a line with dashes is considered its own multiline event, then it should be excluded by exclude_lines. Something like this:

multiline.pattern: '(^[A-Z]+: |^-+$)'

[swright-eti (Steve)]

Thanks. I ended up using mutate in Logstash config.

mutate {
    gsub => [
        # Replace the line of dashes
        "message", "-+$", ""
    ]
}