Multiline pattern not working in filebeat input_type -filestream

[garry12]

I want to make multiline pattern work but the snipped I used below is working only when I change input_type: log and NOT working with input_type:filesteam can someone please help me what I should change to make multiline work?

filebeat.inputs:
- type: filestream
enabled: true
paths:
- /.pss_filecore/2022/*/*/*/*.txt
multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
multiline.max_lines: 3500 output.logstash:
hosts: ["10.10.10.113:5044"]

[stephenb (Stephen Brown)]

There is new syntax for multiline with the new filestream and 8.x.

Basically you're using the old syntax and your multiline is being ignored. You need to put it under the new parsers syntax

parsers

[garry12]

Hey, Yes I was able to follow that I need to use parsers but I am not able to figure out what exactly should I use to parse my text log files ( .txt) If you could help that would be great like I tried using - just wanted to combine 3500 lines together but this is not working please let me know what I can change here to make this work..thanks

filebeat.inputs:
- type: filestream
enabled: true
paths:
- /.pss_filecore/2022/*/*/*/*.txt
parsers:
- multiline:
    type: counter
    lines_count: 3500
output.logstash:
hosts: ["10.10.10.113:5044"]

[stephenb (Stephen Brown)]

Think it's
count_lines
Not
lines_count

Per docs

[garry12]

well its not working with either of those lines_count and count_lines but other confusion I have is the multiline format described in this article is is like -

parsers:
- multiline:
    type: counter
    lines_count: 3500

and in this articles its like this -

multiline.type: pattern
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

so which one I should use or why they are different if both achieve the same goal ?

[garry12]

Hey, good news I got it working after few experiments one of which worked is basically combining the two snippets I posted - So in short I ended up writing parser but properties to match my use case for multiline

filebeat.inputs:

- type: filestream
  enabled: true
  id: id1
  paths:
    - /.pss_filecore/2022/06/*/*/*.txt

  parsers:
  -  multiline:
       pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}'
       negate: true
       match: after
       max_lines: 3500
output.logstash:
  hosts: ["10.10.10.140:5044"]