My system is logging postfix logs (via Filebeat -> Logstash -> ElasticSearch 5.X)
I configured postfix patterns in logstash, so every fields are usable and searchable in kibana.
My logs file looks like below :
line 1 : Jun 25 09:20:29 server postfix/smtpd: D920718033: client=test.local[10.3.45.20]
line 2 : Jun 25 09:20:29 server postfix/cleanup: D920718033: message-id=558BABBD.email@example.com
line 3 : Jun 25 09:20:29 server postfix/qmgr: D920718033: firstname.lastname@example.org, size=28828, nrcpt=1 (queue active)
line 4 : Jun 25 09:20:30 server postfix/pipe: D920718033: email@example.com relay=yyy.fr, delay=0.13, delays=0.07/0.01/0/0.06, dsn=2.0.0, status=sent (delivered via yyy.fr service)
line 5 : Jun 25 09:20:30 server postfix/qmgr: D920718033: removed
As we cannot have multiline input in filebeat for this kind of postfix logs, i would like to group them by the unique postfix_queueid for example, that is present on each line of the smtp transaction. How can we achieve this ? with elasticsearch aggregation ?
I would like for example to search every postfix_status=sent (matching only the line 4) and retrieve automatically its postfix_queueid (ie : D920718033), to display all the logs lines containing this queueid
Is is possible to get this search in 1 step ?
Or get a visualization or graph (via x-pack) to
- search for a specific status (sent, bounce.. whatever)
- show all entrys with the qid of thoses lines
- built the matching graphs with the input of the qid entrys
Thanks in advance for your support.