Multiple distinct multiline patterns; same input file?

[paulh_irl (Paul Harrington)]

Hi all,

Wondering what is the best approach to take when dealing with input log files, where the files contain more than one multiline pattern of interest, and where the patterns are of quite different formats?

For example, multiline pattern 1:

 Model Metrics Type: Regression
  Description: Metrics reported on training frame
  MSE: 0.030414708
  RMSE: 0.17439812

Pattern 2:

 Scoring History:
            Timestamp   Duration Number of Trees Training RMSE Training MAE Training Deviance
  2018-09-10 22:08:46  0.309 sec               0       0.49995      0.49994           0.24995
  2018-09-10 22:08:46  0.548 sec               1       0.44996      0.44995           0.20246
  2018-09-10 22:08:46  0.588 sec               2       0.40496      0.40495           0.16399

(The 2 patterns in this example are edited for brevity. Generally speaking, they are longer than shown, but are of a fixed number of lines)

Given that the different patterns are in the same input file, are we realistically needing more than one instance of filebeat to process them? From what I've read, using 2 prospectors on the same input file can lead to incomplete results. Or would we be better off with a custom module to do some pre/post processing of the input file?

Appreciate any guidance. Many thanks.

[Christian_Dahlqvist]

Based on these two examples it looks like the first line does not start with a space while all following lines do, so you should be able to use something similar to the Java stack traces example in the documentation:

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

[paulh_irl (Paul Harrington)]

Many thanks Christian. Based on a further look at the input file, I believe we can use the lines which you see with leading spaces, by processing them on a single-line basis with different match patterns.

Still wondering though, say if we did find a case where we had 2 multiline segments, and no way to match both, does that require 2 instances of filebeat?

Thanks again for answer.

[Christian_Dahlqvist]

That depends on the pattern as you have a good amount of flexibility regarding the regular expression you can use.

[Alex_Scoble]

You can have a single regex that has multiple patterns by splitting the patterns with |...

So like

'^Model\s+Metrics\s+Type:|^Scoring\s+History:'

For your examples...(can't remember if : needs to be escaped in a regex or not, would have to look it up)

--Alex