Filebeat filestream with pipeline and multiline

[das]

I have a log format I cannot change that leads into multiline messages. I have a Ingest Pipeline set up in Kibana that works just fine on sample records. My problem is that My multiline parser seems to be ignored (at least in error) when I attempt to configure both.

Here's what my filebeat configuration looks like:

filebeat.inputs:
  - type: filestream
    enabled: true
    id: my_env
    index: my_env
    pipeline: mycustompipeline
    paths:
      - /opt/log-mounts/my_env/**/output.log
    fields:
      environment: my_env
    multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
      negate: true
      match: after

Log messages are output based on this configuration from Log4J2 in a Java project (again, I can't just add the JSON appender as it's a COTS app that won't allow it):

<Pattern>%d [%20.20t] [%10.10X{key1}] [%20.20X{key2}] [%20.20X{key3}] (%30.30c{3}) %-5p %X{key4} %X{key5} %X{key6} - %m%n</Pattern>

What do I need to do to have the multiline parse correctly AND run before the pipeline?

[das]

I got most stuff working:

filebeat.inputs:
  - type: filestream
    enabled: true
    id: my_env
    index: my_env
    paths:
      - /opt/log-mounts/my_env/**/output.log
    fields:
      environment: my_env

    multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
    multiline.negate: true
    mutiline.match: after

    pipeline: mycustompipeline

I just need to get it to include blank lines in multieline parsing.

[stephenb (Stephen Brown)]

Hi @das What version filebeat, that is not the correct syntax for multiline using - type: filestream

That is the old / previous syntax that goes with the deprectation - type: log input

@das

    multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
    multiline.negate: true
    mutiline.match: after

See here

Should look like

parsers:
- multiline:
    type: pattern
    pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
    negate: true
    match: after

[das]

I was able to fix this! What you have there is correct!
I got backed up and forgot to post the final solution I came up with:

filebeat.inputs:
  - type: filestream
    enabled: true
    id: my_env
    index: my_env
    paths:
      - /opt/log-mounts/my_env/**/output.log
    fields:
      environment: my_env
    parsers:
      - multiline:
          pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ \['
          negate: true
          match: after
    pipeline: mycustompipeline

Once I got everything squared away and realizing the documentation I was reading and even ChatGPT were not the right version, this all flowed together really fast. I also was able to verify that the multiline parser is picking up blank lines appropriate as well.

My lesson was most definitely:
MAKE SURE YOU ARE ON THE RIGHT DOCUMENTATION VERSION

There's some weirdness with the app where it seems to ignore this at random, but that's not an issue for here:

%X{key4} %X{key5} %X{key6}