Grouping postfix queueid

[toger (Tony OGER)]

My system is logging postfix logs (via Filebeat -> Logstash -> ElasticSearch 5.X)

I configured postfix patterns in logstash, so every fields are usable and searchable in kibana.
My logs file looks like below :

line 1 : Jun 25 09:20:29 server postfix/smtpd[18999]: D920718033: client=test.local[10.3.45.20]
line 2 : Jun 25 09:20:29 server postfix/cleanup[19004]: D920718033: message-id=558BABBD.7010601@yyy.fr
line 3 : Jun 25 09:20:29 server postfix/qmgr[3759]: D920718033: from=xxx@yyy.fr, size=28828, nrcpt=1 (queue active)
line 4 : Jun 25 09:20:30 server postfix/pipe[19005]: D920718033: to=xxx@yyy.fr relay=yyy.fr, delay=0.13, delays=0.07/0.01/0/0.06, dsn=2.0.0, status=sent (delivered via yyy.fr service)
line 5 : Jun 25 09:20:30 server postfix/qmgr[3759]: D920718033: removed

As we cannot have multiline input in filebeat for this kind of postfix logs, i would like to group them by the unique postfix_queueid for example, that is present on each line of the smtp transaction. How can we achieve this ? with elasticsearch aggregation ?

I would like for example to search every postfix_status=sent (matching only the line 4) and retrieve automatically its postfix_queueid (ie : D920718033), to display all the logs lines containing this queueid
Is is possible to get this search in 1 step ?

Or get a visualization or graph (via x-pack) to

1. search for a specific status (sent, bounce.. whatever)
2. show all entrys with the qid of thoses lines
3. built the matching graphs with the input of the qid entrys

Thanks in advance for your support.

[warkolm (Mark Walkom)]

If all the events flow through one processing pipeline, you can use an aggregation filter in LS.
Otherwise you could use Graph to link them once in ES.

[toger (Tony OGER)]

Yes all events comes through logstash. Sources comes from several filebeat instances with each one their own index.
I have found this aggregation filter that could suit my needs : https://github.com/topmedia/logstash-postfix/blob/master/etc/logstash/conf.d/51-filter-postfix-aggregate.conf

I think i will try to use Graph also to see how correlation between queueids will be displayed, and perhaps the event aggregation groupby queueid won't be necessary.

Thanks