Multiline, Nonconsecutive

[Lazarbeam (David Lazar)]

All --

We have logs that look like the following:

8 - {5587} Data1
8 - {5587} Data2
8 - {18573} Data1
8 - {18573} Data2
8 - {18551} Data1
8 - {18551} Data2
8 - {9610} Data1
8 - {18574} Data1
8 - {9610} Data2
8 - {18574} Data2
8 - {7167} Data1
8 - {5587} Data1
8 - {7167} Data2
8 - {5587} Data2

Each event records 2 lines of logs - one with Data1, another with Data2. The process ID in curly braces is what joins them together. However, the lines are not guaranteed to be consecutive, and the proceess ID can be reused later in the log.

I'm struggling to get Filebeat to prospect the data when:

• Lines are not consecutive
• Each log line would use an identical regex as a pattern match
• The process ID which identifies logs from the same even can occur further in the log - I'm less concerned about this because I can define a max lines of logs to match.

In a general sense, is this even possible?

My last attempt landed me at:

multiline.pattern: '^\d \- \{\d+\}'
multiline.match: after
multiline.maxlines: 2

But I imagine that would match the very next line, regardless of whether the process ID matches.

Thanks in advance!

[andrewkroh]

@Lazarbeam

Lines are not consecutive

This is not something that multiline is designed for. It's only for joining consecutive lines.

You should look at using Filebeat sending to Logstash. And in LS using the aggregate filter.