Exclude data before multiline?

Elastic Stack Beats
1

I use ELK stack + filebeat to collect logs. Log flows in below order.

filebeat(1.3.0) -> logstash(5.0.0 alpha5) -> elasticsearch(5.0.0alpha5)

I also use multiline in filebeat to group certain lines.

Problem I am having is multiline worked perfectly until an error log (Hoge : Error : Data is wrong) slipped in between the lines. Below is the sample log.

[09:00:41.219593] test 0
[09:00:41.219593] test2 10231
[09:00:41.219718] test3 0
[09:00:41.219718] test4 3679
[09:00:41.219718] test5 0
[09:00:41.219718] test6 924
[09:00:41.219718]
[09:00:41.219718] test7 0
[09:00:41.219718] Hoge : Error : Data is wrong
[09:00:41.220843] test8 8770
[09:00:41.220843] test9 0
[09:00:41.220843] test10 3458
[09:00:41.220843] test11 0
[09:00:41.220843] test12 2372

Is there a way to exclude line before multiline takes place in the filebeat?
Document clearly defines exclude takes place after the multiline but I want to implement this.

2

There is already an open issue for that here: https://github.com/elastic/beats/issues/1940 Currently exclude_lines is a processor and multiline is a reader. Processors are only applied after the reader finished. It would required some reorganisation. Happy to hear some suggestions on how could solve this issue.

3

@ruflin

Understood . Thank you for the reference.

4

This topic was automatically closed after 21 days. New replies are no longer allowed.