Filebeat multiline by Queue ID

Elastic Stack Beats
1

Hello Community,
I want to filter Postfix log lines and tie them together by Postfix Queue ID.

Currently for this purpose I am using Logstash multiline filter in combination with its parameter "stream_identity".
%{queueid} is a grok pattern.

multiline {
stream_identity => "%{queueid}"
pattern => "."
what => previous
periodic_flush => true
max_age => 120
}

This configuration ties all loglines with the same Queue ID togehter.

Is there any possibility to get the same result also with the new Filebeat multiline filter?

Thanks in advance!

2

No, because Filebeat can't parse log lines.

The whole idea is flawed. All log messages pertaining to a Postfix queue id might accumulate over several days. Attempting to merge them all into a single event using any kind of multiline filter is a mistake.

1 Like
3

Hello and thanks for your reply,
yes they might accumulate over several days but 95%+ are go through in about 2 seconds. For the worst cases I have put "max_age => 120".

Thanks anyway.

Joey

4

why combine all messages into one. Isn't the idea of indexing these into elasticsearch and kibana to enable users to filter based on parsed schema?

5

Hello,
the idea is to combine all fields parsed by logstash into one single event.

This let me run queries like this one in kibana:
sender:"you@example.com" AND recipient:"recipient@example2.com" AND amavis_action:"pass"

Without knowing the postfix queue ID.

And also this is only possible if you have all the fields in one single event.

6

Hello Magnus, you say using the multiline option to tie events is a mistake. Are there other alternatives? especially in a multi-thread environment? Honestly if I find no other altenatives, I'll look for other solutions, like parsing the json format before data is indexed. Grok is really limited option..

7

How will parsing json help in your use-case. Btw. simple json supported added to filebeat 5.0 alpha1).

The feature you're asking for is join/correlation (unless you can use filters in kibana). Not sure this is supported by logstash though.

8

Using logstash-logback-encoder for instance allows to encode messages logged in JSON, and then send them via a TCP socket to Logstash. The JSON content would be already structured..and yes I've been told the join option is not supported by Logstash. It is really a shame that cannot be done via grok.

9

Using logstash-logback-encoder for instance allows to encode messages logged in JSON, and then send them via a TCP socket to Logstash. The JSON content would be already structured..

Yes, that's preferred to parsing text.

and yes I've been told the join option is not supported by Logstash.

It depends. Have you looked at the aggregate filter?

It is really a shame that cannot be done via grok.

The grok filter extracts fields from text using regular expressions. That's simply unrelated to what you're trying to do.

10

Thanks for answering. It helps.

Actually, my filter workers are set to multiple which makes it impossible to use any of the multiline or aggregate filters. Can't lower the number to 1. And the reason why I can't use logstash-logback-encoder for the moment is that I am using my server to archive my logs, so I need to receive the logs in the original format and not JSON structured.

I really wish someone could help me and shed light on this because I am really at loss at how to tie together the related events with filters or scripts if it is feasable, to display them on KIBANA. I've been looking for a solution for months now. Sorry I am using this page to get answers, I am unable to get them elsewhere.

11

I'd keep the Logstash parsing simple and have it emit low-level events that correspond to the actual events. Then I'd feed those, probably via a broker, to a service (possibly Logstash again) that correlates events and emits high-level events with all information about e.g. an email transaction.