# Filebeat multiline by Queue ID
**URL:** https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406
**Category:** Beats
**Tags:** filebeat
**Created:** [April 14, 2016, 3:01pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406 "2016-04-14T15:01:55Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Joey](https://discuss.elastic.co/u/Joey)
#### Post date: [April 14, 2016, 3:01pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/1 "2016-04-14T15:01:55Z")
Hello Community,
I want to filter Postfix log lines and tie them together by Postfix Queue ID.
Currently for this purpose I am using Logstash multiline filter in combination with its parameter "stream\_identity".
%{queueid} is a grok pattern.
> multiline {
> stream\_identity =\> "%{queueid}"
> pattern =\> "."
> what =\> previous
> periodic\_flush =\> true
> max\_age =\> 120
> }
This configuration ties all loglines with the same Queue ID togehter.
Is there any possibility to get the same result also with the new Filebeat multiline filter?
Thanks in advance!
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [April 14, 2016, 5:50pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/2 "2016-04-14T17:50:06Z")
No, because Filebeat can't parse log lines.
The whole idea is flawed. All log messages pertaining to a Postfix queue id might accumulate over several days. Attempting to merge them all into a single event using any kind of multiline filter is a mistake.
---
### Author:  [@Joey](https://discuss.elastic.co/u/Joey)
#### Post date: [April 15, 2016, 6:23am UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/3 "2016-04-15T06:23:20Z")
> [@Joey](#):
>
> max\_age =\> 120
Hello and thanks for your reply,
yes they might accumulate over several days but 95%+ are go through in about 2 seconds. For the worst cases I have put "max\_age =\> 120".
Thanks anyway.
Joey
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 15, 2016, 1:56pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/4 "2016-04-15T13:56:24Z")
why combine all messages into one. Isn't the idea of indexing these into elasticsearch and kibana to enable users to filter based on parsed schema?
---
### Author:  [@Joey](https://discuss.elastic.co/u/Joey)
#### Post date: [April 18, 2016, 10:26am UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/5 "2016-04-18T10:26:25Z")
> [@steffens](#):
>
> t
Hello,
the idea is to combine all fields parsed by logstash into one single event.
This let me run queries like this one in kibana:
[sender:"you@example.com](mailto:sender:%22you@example.com)" AND [recipient:"recipient@example2.com](mailto:recipient:%22recipient@example2.com)" AND amavis\_action:"pass"
Without knowing the postfix queue ID.
And also this is only possible if you have all the fields in one single event.
---
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [April 29, 2016, 9:33am UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/6 "2016-04-29T09:33:49Z")
Hello Magnus, you say using the multiline option to tie events is a mistake. Are there other alternatives? especially in a multi-thread environment? Honestly if I find no other altenatives, I'll look for other solutions, like parsing the json format before data is indexed. Grok is really limited option..
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 29, 2016, 12:04pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/7 "2016-04-29T12:04:09Z")
How will parsing json help in your use-case. Btw. simple json supported added to filebeat 5.0 alpha1).
The feature you're asking for is join/correlation (unless you can use filters in kibana). Not sure this is supported by logstash though.
---
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [April 29, 2016, 12:53pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/8 "2016-04-29T12:53:16Z")
Using logstash-logback-encoder for instance allows to encode messages logged in JSON, and then send them via a TCP socket to Logstash. The JSON content would be already structured..and yes I've been told the join option is not supported by Logstash. It is really a shame that cannot be done via grok.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [April 30, 2016, 2:48pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/9 "2016-04-30T14:48:44Z")
> Using logstash-logback-encoder for instance allows to encode messages logged in JSON, and then send them via a TCP socket to Logstash. The JSON content would be already structured..
Yes, that's preferred to parsing text.
> and yes I've been told the join option is not supported by Logstash.
It depends. Have you looked at the [aggregate filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html)?
> It is really a shame that cannot be done via grok.
The grok filter extracts fields from text using regular expressions. That's simply unrelated to what you're trying to do.
---
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [April 30, 2016, 5:59pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/10 "2016-04-30T17:59:36Z")
Thanks for answering. It helps.
Actually, my filter workers are set to multiple which makes it impossible to use any of the multiline or aggregate filters. Can't lower the number to 1. And the reason why I can't use logstash-logback-encoder for the moment is that I am using my server to archive my logs, so I need to receive the logs in the original format and not JSON structured.
I really wish someone could help me and shed light on this because I am really at loss at how to tie together the related events with filters or scripts if it is feasable, to display them on KIBANA. I've been looking for a solution for months now. Sorry I am using this page to get answers, I am unable to get them elsewhere.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 3, 2016, 5:59am UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/11 "2016-05-03T05:59:13Z")
I'd keep the Logstash parsing simple and have it emit low-level events that correspond to the actual events. Then I'd feed those, probably via a broker, to a service (possibly Logstash again) that correlates events and emits high-level events with all information about e.g. an email transaction.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 9:52pm UTC](https://discuss.elastic.co/t/filebeat-multiline-by-queue-id/47406/12 "2017-07-05T21:52:31Z")