# Multiple distinct multiline patterns; same input file?

**URL:** <https://discuss.elastic.co/t/multiple-distinct-multiline-patterns-same-input-file/148027>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 10, 2018, 10:36pm UTC](https://discuss.elastic.co/t/multiple-distinct-multiline-patterns-same-input-file/148027 "2018-09-10T22:36:08Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![paulh\_irl](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@paulh\_irl](https://discuss.elastic.co/u/paulh_irl)\
**Post date:** [September 10, 2018, 10:36pm UTC](https://discuss.elastic.co/t/multiple-distinct-multiline-patterns-same-input-file/148027/1 "2018-09-10T22:36:08Z")

</div>

Hi all,

Wondering what is the best approach to take when dealing with input log files, where the files contain more than one multiline pattern of interest, and where the patterns are of quite different formats?

For example, **multiline pattern 1:**

```
 Model Metrics Type: Regression
  Description: Metrics reported on training frame
  MSE: 0.030414708
  RMSE: 0.17439812

```

**Pattern 2:**

```
 Scoring History:
            Timestamp Duration Number of Trees Training RMSE Training MAE Training Deviance
  2018-09-10 22:08:46 0.309 sec 0 0.49995 0.49994 0.24995
  2018-09-10 22:08:46 0.548 sec 1 0.44996 0.44995 0.20246
  2018-09-10 22:08:46 0.588 sec 2 0.40496 0.40495 0.16399

```

(The 2 patterns in this example are edited for brevity. Generally speaking, they are longer than shown, but are of a fixed number of lines)

Given that the different patterns are in the same input file, are we realistically needing more than one instance of filebeat to process them? From what I've read, using 2 prospectors on the same input file can lead to incomplete results. Or would we be better off with a custom module to do some pre/post processing of the input file?

Appreciate any guidance. Many thanks.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 11, 2018, 5:40am UTC](https://discuss.elastic.co/t/multiple-distinct-multiline-patterns-same-input-file/148027/2 "2018-09-11T05:40:35Z")

</div>

Based on these two examples it looks like the first line does not start with a space while all following lines do, so you should be able to use something similar to the [Java stack traces example in the documentation](https://www.elastic.co/guide/en/beats/filebeat/6.4/_examples_of_multiline_configuration.html#_java_stack_traces):

```auto
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

```

---

<div class="post-metadata">

**Author:** ![paulh\_irl](https://avatars.discourse-cdn.com/v4/letter/p/13edae/32.png) [@paulh\_irl](https://discuss.elastic.co/u/paulh_irl)\
**Post date:** [September 11, 2018, 9:04am UTC](https://discuss.elastic.co/t/multiple-distinct-multiline-patterns-same-input-file/148027/3 "2018-09-11T09:04:31Z")

</div>

Many thanks Christian. Based on a further look at the input file, I believe we can use the lines which you see with leading spaces, by processing them on a single-line basis with different match patterns.

Still wondering though, say if we did find a case where we had 2 multiline segments, and no way to match both, does that require 2 instances of filebeat?

Thanks again for answer.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 11, 2018, 9:06am UTC](https://discuss.elastic.co/t/multiple-distinct-multiline-patterns-same-input-file/148027/4 "2018-09-11T09:06:50Z")

</div>

That depends on the pattern as you have a good amount of flexibility regarding the regular expression you can use.

---

<div class="post-metadata">

**Author:** ![Alex\_Scoble](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_scoble/32/38142_2.png) [@Alex\_Scoble](https://discuss.elastic.co/u/Alex_Scoble)\
**Post date:** [September 11, 2018, 10:56pm UTC](https://discuss.elastic.co/t/multiple-distinct-multiline-patterns-same-input-file/148027/5 "2018-09-11T22:56:03Z")

</div>

You can have a single regex that has multiple patterns by splitting the patterns with |...

So like

```
'^Model\s+Metrics\s+Type:|^Scoring\s+History:'

```

For your examples...(can't remember if : needs to be escaped in a regex or not, would have to look it up)

--Alex

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2018, 10:56pm UTC](https://discuss.elastic.co/t/multiple-distinct-multiline-patterns-same-input-file/148027/6 "2018-10-09T22:56:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.