2 questions about multiline patterns

Elastic Stack Beats
1

hello,

using Filebeat 6.7.1

I have a couple of questions about the usage of multiline options for the "log" input.

Question 1: is there a way to apply include_lines before multiline?

The log file I am reading has a format like this

key1 = value1
key2 = value2
...
key10 = value10
*** END ***
key1 = value1
key2 = value2
...
key10 = value10
*** END ***
key1 = value1
...

and I would like to merge into a single line those for keys 2 and 5, for example.
I thought, naively, something like this

include_lines = ['^\*\*\*', '^key2', 'key5' ]
multiline.pattern = '^\*\*\*'
multiline.negate = true
multiline.match = before

would give me this

{
    key2 = value2\nkey5 = value5\n *** END ***
}
{
    key2 = value2\nkey5 = value5\n *** END ***
}
{
    key2 = value2\nkey5 = value5\n *** END ***
}

However, I read this in the documentation:

include_lines
A list of regular expressions [...]
If multiline settings also specified, each multiline message is combined into a single line before the lines are filtered by include_lines.

Is there some trick to filter before merging?

Question 2: does multiline mix lines from different input files?

If configuration is like this

filebeat.inputs:
- type: log
  paths:
    - /var/log/app/logs*   # <= unavoidable !!
  multiline.pattern = ... whatever ...
  multiline.negate = ...
  multiline.match = ...

Will it merge only lines from the same input file? Or it could eventually merge together lines coming from different input files?
If the latter, is it possible to prevent that?

Thanks a lot in advance.
Cheers,
Jose

2

Question 1: is there a way to apply include_lines before multiline?

No, at the moment there isn't. This could perhaps be an enhancement though. If you have a GitHub account, please create an enhancement issue here: https://github.com/elastic/beats/issues/new?template=feature-request.md. That way you can explain your use case in detail and follow along with any discussion and updates. If you don't have a GitHub account, let me know and I'll make an issue for you.

Question 2: does multiline mix lines from different input files?

It does not. Multiline operates at the harvester level. One harvester corresponds to a single log file.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.