Distinct LS port per prospector for multiline

[g.le]

Hello everyone!
My current filebeat configuration reads logs from two different directories. Each directory contains logs of distinct formats, therefore I have two log formats.
For each directory, I have specified a filebeat prospector and I have assigned distinct tags.

Apparently, I cannot assign distinct shipping destinations (note: I am mostly interested in distinct ports) per prospector. This means that Logstash will retrieve all logs via the same port and will attempt to apply a single multiline pattern for the initial parsing of logs.

Unfortunately, this will not work in my case since two patterns are required (one for each log format). What makes it worse is that the pattern for one of the log formats is rather unknown/variable.

Any suggestions? Should I have to go with two filebeat installations?

Thank you!

[andrewkroh]

You cannot route events to different outputs in Beats, but it sounds like you may not need to. You can do the multiline aggregation in Beats rather than using a codec in Logstash.

[g.le]

Hello @andrewkroh,
Thank you for the prompt response.

As far as I understand, you suggest using the multiline option of Filebeat (instead of the multiline pattern of beats input plugin of Logstash) because it can be specified per prospector.
Am I right?

Thank you!

[andrewkroh]

I recommend doing the multiline inside of Filebeat so that you don't have to worry about the multline codec inside of Logstash mixing up data coming from multiple files. Like the warning in the Filebeat documentation states:

Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.

The multiline configuration is specified per prospector and those settings apply to all the files scanned by that prospector. But the multiline state is stored on a per file basis in Filebeat's memory.

So if you have files with different multline patterns you would use config like:

filebeat.prospectors:
- paths: ['/appA/logs/*']
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  tags: [appA]
- paths: ['/appB/logs/*']
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  tags: [appB]

[g.le]

@andrewkroh: Thank you for the suggestion!

I am quite confused though, as I use the exact same pattern in Filebeat prospector as the one I used to in Logstash but apparently that does not work.

If that helps:

What I had in Logstash:

---

codec => multiline {
  pattern => "<message"
  negate => "true"
  what => "previous"
}

What I now have in Filebeat:

---

filebeat.prospectors:
- input_type: log
  paths:
    - 'C:\myLogs\*'
  multiline.pattern: '<message'
  multiline.negate: true
  multiline.match: before