Filebeat multiline pattern from 'abbbc' to 'ac'

Elastic Stack Beats
,
1

Hello all,

The last few days I have been struggling with a multiline pattern. My multiline log messages have a pattern like 'abbbc'. I don't need the b-parts to be send to LogStash. So basically what I want to accomplish is to convert a multiline log message like

a
b
b
b
c

into

ac

Unfortunately the "exclude_lines" parameter is handled after the "multiline.pattern" parameter in the filebeat.yml.

Any ideas on how to crack this case?

2

How about using Filebeat just for the exclude_lines part and having it send the rest of the lines to Logstash as single events? And then using the Logstash multiline codec on the beats input?

3

That could be a solution. However I'm not in control of LogStash. I work in a fairly large company and each team has certain responsibilities. LogStash is a service from another team. So I'd rather solve this within a Filebeat configuration (if possible).

4

I see. Then how about using two Filebeat instances? The first one could read from your log file, use exclude_lines to exclude the b lines, and use the file output to write to intermediate log files. The second one could read from this intermediate log file, use multiline, and send the resulting events to Logstash.

You would probably want to write intermediate log files by day or by hour so you can have something come around and clean them up periodically.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.