Problem to copy a field from one event to another event

[abu.sayeed]

Two event's of postfix-mail log fields are below.:

source: /var/log/mail.log
postfix_queueid:24C4C681F19
host: mail
program: postfix/amavis/smtp
postfix_to: jhon@gmail.com
time: 10:53:30
postfix_status: sent
postfix_delay: 2.1
postfix_relay_ip: 127.0.0.1
postfix_delay_in_qmgr: 0.01

source: /var/log/mail.log
postfix_queueid: 24C4C681F19
postfix_size: 8209
host: mail
time: 10:53:30
program: postfix/qmgr
postfix_from: salam@gmail.com

I want to copy a field postfix_from of program:postfix/qmgr To program:postfix/amavis/smtp if their both postfix_queueid is same. otherwise no copy

I need a complete information. like from, to, status etc in kibana dashboard.

Thanks all

[pjanzen (Paul Janzen)]

As far as I know this cannot be done in logstash. You want to combine events that match the postfix_id. In order to accomplish this you need to temporarily store your events and query those while processing event and if you have a match push them to elasticsearch. Pushing the temp data to ES will not help as you will always have a lag between storing the data and when the data is available for your next query. This lag results in a failed query inside your filter part.

You can do a update on an existing document if you use the postfix_id as document id for example but you will get very poor performance results if you go that way.

You might be better of into looking into filebeat for this where you can apply a multi-line filter in side filbeat or make sure postfix only sends 1 log line per event..

[pjanzen (Paul Janzen)]

There might be way to do what you need using other means.

Have a read - here - this might provide what you are looking for.

(full disclaimer, I do not support that nor do I have any knowlage about it)