Filebeat vs logstash for handling multiline

Elastic Stack Logstash
1

Hi,

We have 40 instances sending logs parallely to centralized logstash via filebeat. We have tried multiline codec with logstash and filebeat both as well. Both seems to be working.

But we see few issues with logstash like filebeat queues files when log rotation happens during high volume duration. (we run logstash with default worker i.e 1).

Is there any recommendation for handling multiline messages should we use filebeat or logstash?

2

Performing multiline processing as close to the source as possible is generally preferable, so I would recommend doing it in Filebeat. This will allow you to send data either to a message queue or load balance across multiple Logstash instances without having to worry about lines that are supposed to be merged at a later stage get split up.

3

But the multiline possibilities in filebeat is only the one that merges specific line endings right? The one meant for exceptions and the like?

Is there a way to set this up for matching a start-line + end-line (sorry if this hijacks this post)

4

Thanks for your swift response :slight_smile: Because i don't see any discussion on this topic. So just wanted to know more details on this comparison.

5

multiline for filebeat works for start line as well. For application traceback it works. Just a slight difference in syntax from logstash grok. Will post you some examples soon for python traceback.

6

That would be great. Thanks

It's just something like

Event_starts
EventContent
EventContent
Event_ends

Then matching the start and end words (or just the start) to merge into a single event from which fields can be extracted in the content lines

Just haven't 'gotten' the configs yet I guess

7

Inspired by your suggestion on filbert/multiline, I've dug a bit deeper, and found that it can, as you said, merge multiline based on a start-line, if you use it something like this

multiline
- pattern: "start-line-pattern"
- negate: true
- match: after

Only slight thing i haven't worked out, is how to make it match the end of an event.

This is because this multiline configuration will continue waiting for the next "start-line-pattern", at which it will then emit the current event, and start another.. But then the last event will not be emitted, until the given timeout is reached.

Is there anyway to specify a flush pattern, ending the multiline?

something like:

multiline
- pattern: "start-line-pattern"
- negate: true
- match: after
- flush-pattern: "end-line-pattern"

That would be pretty cool :slight_smile:

edit: I filed a suggestion for this on the beats GitHub page (Filebeat suggestion - allowing for something like multiline.flush_pattern? · Issue #3964 · elastic/beats · GitHub)
Please let me know if this is possible already :slight_smile:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.