# Try to Parsing Postfix Log Using Logstash **URL:** \ **Category:** Logstash\ **Created:** [November 4, 2021, 10:39am UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397 "2021-11-04T10:39:25Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![RichYaNa](https://avatars.discourse-cdn.com/v4/letter/r/ba8739/32.png) [@RichYaNa](https://discuss.elastic.co/u/RichYaNa)\ **Post date:** [November 4, 2021, 10:39am UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397/1 "2021-11-04T10:39:25Z")
Hi, I'm currently parsing postfix logs using Logstash, you can check my config file [here](https://github.com/RichiMaulana/Parsing-Postfix-Log). and here is a sample log that I will parse: ```auto Nov 4 08:51:20 mail /postfix-script[XPIDX]: the Postfix mail system is running: PID: XPIDX Nov 4 08:51:29 mail postfix/postscreen[]: CONNECT from [SOMEONEIPADDRESS]:PORT to [MYSERVERIPADDRESS]:PORT Nov 4 08:51:29 mail postfix/postscreen[XPIDX]: ALLOWLISTED [SOMEONEIPADDRESS]:PORT Nov 4 08:51:29 mail postfix/smtpd[XPIDX]: connect from my.server.hostname[MYSERVERIPADDRESS] Nov 4 08:51:29 mail postfix/smtpd[XPIDX]: NOQUEUE: filter: RCPT from my.server.hostname[MYSERVERIPADDRESS]: : Sender address triggers FILTER smtp-amavis:[127.0.0.1]:PORT; from= to= proto=ESMTP helo= Nov 4 08:51:29 mail postfix/smtpd[XPIDX]: 2797FC2D92: client=my.server.hostname[MYSERVERIPADDRESS] Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 2797FC2D92: message-id= Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 2797FC2D92: warning: header Subject: THIS IS SUBJECT from my.server.hostname[MYSERVERIPADDRESS]; from= to= proto=ESMTP helo= Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 2797FC2D92: from=, size=1294, nrcpt=1 (queue active) Nov 4 08:51:29 mail postfix/smtpd[XPIDX]: disconnect from my.server.hostname[MYSERVERIPADDRESS] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 Nov 4 08:51:29 mail postfix/dkimmilter/smtpd[10380]: 36BEBC2D93: client=localhost[127.0.0.1] Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 36BEBC2D93: message-id= Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 36BEBC2D93: warning: header Subject: THIS IS SUBJECT from localhost[127.0.0.1]; from= to= proto=ESMTP helo= Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 36BEBC2D93: from=, size=1787, nrcpt=1 (queue active) Nov 4 08:51:29 mail postfix/dkimmilter/smtpd[10380]: disconnect from localhost[127.0.0.1] ehlo=1 mail=2 rcpt=2 data=2 noop=1 quit=1 commands=9 Nov 4 08:51:29 mail postfix/smtp[XPIDX]: 2797FC2D92: to=, relay=127.0.0.1[127.0.0.1]:PORT, delay=0.12, delays=0.01/0/0/0.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10030): 250 2.0.0 Ok: queued as 36BEBC2D93) Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 2797FC2D92: removed Nov 4 08:51:29 mail postfix/amavisd/smtpd[10381]: 58600C2D92: client=localhost[127.0.0.1] Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 58600C2D92: message-id= Nov 4 08:51:29 mail postfix/amavisd/smtpd[10381]: disconnect from localhost[127.0.0.1] ehlo=1 mail=2 rcpt=2 data=2 noop=1 quit=1 commands=9 Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 58600C2D92: from=, size=2161, nrcpt=1 (queue active) Nov 4 08:51:29 mail postfix/smtp[XPIDX]: 36BEBC2D93: to=, relay=127.0.0.1[127.0.0.1]:10032, delay=0.15, delays=0.05/0/0/0.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 58600C2D92) Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 36BEBC2D93: removed Nov 4 08:51:40 mail postfix/smtp[XPIDX]: 58600C2D92: to=, relay=relay.hostname.tld[172.104.50.189]:587, delay=12, delays=0.01/0/10/1.2, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as F13E9BBCF8) Nov 4 08:51:40 mail postfix/qmgr[XPIDX]: 58600C2D92: removed ``` But now I'm stuck because of a problem. The output from logstash is not what I want. Here's an example of the output: ```auto { "_index" : "My_Elasticsearch_Index", "_type" : "_doc", "_id" : "The_Document_1_ID", "_score" : 1.0, "_source" : { "path" : "the/path/to/the/log/filter", "subject" : "This is the subject value should like to be", "postfix_to" : "recipient@domain.tld", "postfix_status" : "sent", "timestamp" : "Nov 2 02:30:22",, "postfix_from" : "sender@domain.tld", "@timestamp" : "2021-11-02T02:30:22.656Z", "program" : "postfix/smtp", "message" : "TheQueueID: to=, relay=The_Relay_Hostname[The_Relay_IP]:PORT, delay=0.15, delays=0.01/0.03/0/0.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[The_MTA_IP]:PORT): 250 2.0.0 Ok: queued as TheNextQueueID)", "postfix_queueid" : "TheQueueID", "postfix_message-id" : null, "tags" : [ "_grok_postfix_success" ] } }, { "_index" : "My_Elasticsearch_Index", "_type" : "_doc", "_id" : "The_Document_2_ID", "_score" : 1.0, "_source" : { "path" : "the/path/to/the/log/filter", "subject" : null, "postfix_to" : "recipient@domain.tld", "postfix_status" : "sent", "timestamp" : "Nov 2 02:30:22",, "postfix_from" : "sender@domain.tld", "@timestamp" : "2021-11-02T02:30:22.656Z", "program" : "postfix/smtp", "message" : "TheQueueID: to=, relay=The_Relay_Hostname[The_Relay_IP]:PORT, delay=0.15, delays=0.01/0.03/0/0.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[The_MTA_IP]:PORT): 250 2.0.0 Ok: queued as TheNextQueueID)", "postfix_queueid" : "TheQueueID", "postfix_message-id" : "Some_Random_Numbers_and_Word_that_define_the_message_ID@domain.tld",, "tags" : [ "_grok_postfix_success" ] } }, ``` I'm currently parsing postfix logs using Logstash, you can check my config file [here](https://github.com/RichiMaulana/Parsing-Postfix-Log). and here is a sample log that I will parse: ```auto Nov 4 08:51:20 mail /postfix-script[XPIDX]: the Postfix mail system is running: PID: XPIDX Nov 4 08:51:29 mail postfix/postscreen[]: CONNECT from [SOMEONEIPADDRESS]:PORT to [MYSERVERIPADDRESS]:PORT Nov 4 08:51:29 mail postfix/postscreen[XPIDX]: ALLOWLISTED [SOMEONEIPADDRESS]:PORT Nov 4 08:51:29 mail postfix/smtpd[XPIDX]: connect from my.server.hostname[MYSERVERIPADDRESS] Nov 4 08:51:29 mail postfix/smtpd[XPIDX]: NOQUEUE: filter: RCPT from my.server.hostname[MYSERVERIPADDRESS]: : Sender address triggers FILTER smtp-amavis:[127.0.0.1]:PORT; from= to= proto=ESMTP helo= Nov 4 08:51:29 mail postfix/smtpd[XPIDX]: 2797FC2D92: client=my.server.hostname[MYSERVERIPADDRESS] Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 2797FC2D92: message-id= Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 2797FC2D92: warning: header Subject: THIS IS SUBJECT from my.server.hostname[MYSERVERIPADDRESS]; from= to= proto=ESMTP helo= Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 2797FC2D92: from=, size=1294, nrcpt=1 (queue active) Nov 4 08:51:29 mail postfix/smtpd[XPIDX]: disconnect from my.server.hostname[MYSERVERIPADDRESS] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 Nov 4 08:51:29 mail postfix/dkimmilter/smtpd[10380]: 36BEBC2D93: client=localhost[127.0.0.1] Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 36BEBC2D93: message-id= Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 36BEBC2D93: warning: header Subject: THIS IS SUBJECT from localhost[127.0.0.1]; from= to= proto=ESMTP helo= Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 36BEBC2D93: from=, size=1787, nrcpt=1 (queue active) Nov 4 08:51:29 mail postfix/dkimmilter/smtpd[10380]: disconnect from localhost[127.0.0.1] ehlo=1 mail=2 rcpt=2 data=2 noop=1 quit=1 commands=9 Nov 4 08:51:29 mail postfix/smtp[XPIDX]: 2797FC2D92: to=, relay=127.0.0.1[127.0.0.1]:PORT, delay=0.12, delays=0.01/0/0/0.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10030): 250 2.0.0 Ok: queued as 36BEBC2D93) Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 2797FC2D92: removed Nov 4 08:51:29 mail postfix/amavisd/smtpd[10381]: 58600C2D92: client=localhost[127.0.0.1] Nov 4 08:51:29 mail postfix/cleanup[XPIDX]: 58600C2D92: message-id= Nov 4 08:51:29 mail postfix/amavisd/smtpd[10381]: disconnect from localhost[127.0.0.1] ehlo=1 mail=2 rcpt=2 data=2 noop=1 quit=1 commands=9 Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 58600C2D92: from=, size=2161, nrcpt=1 (queue active) Nov 4 08:51:29 mail postfix/smtp[XPIDX]: 36BEBC2D93: to=, relay=127.0.0.1[127.0.0.1]:10032, delay=0.15, delays=0.05/0/0/0.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 58600C2D92) Nov 4 08:51:29 mail postfix/qmgr[XPIDX]: 36BEBC2D93: removed Nov 4 08:51:40 mail postfix/smtp[XPIDX]: 58600C2D92: to=, relay=relay.hostname.tld[172.104.50.189]:587, delay=12, delays=0.01/0/10/1.2, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as F13E9BBCF8) Nov 4 08:51:40 mail postfix/qmgr[XPIDX]: 58600C2D92: removed ``` But now I'm stuck because of a problem. The output from logstash is not what I want. Here's an example of the output: ```auto { "_index" : "My_Elasticsearch_Index", "_type" : "_doc", "_id" : "The_Document_1_ID", "_score" : 1.0, "_source" : { "path" : "the/path/to/the/log/filter", "subject" : "This is the subject value should like to be", "postfix_to" : "recipient@domain.tld", "postfix_status" : "sent", "timestamp" : "Nov 2 02:30:22",, "postfix_from" : "sender@domain.tld", "@timestamp" : "2021-11-02T02:30:22.656Z", "program" : "postfix/smtp", "message" : "TheQueueID: to=, relay=The_Relay_Hostname[The_Relay_IP]:PORT, delay=0.15, delays=0.01/0.03/0/0.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[The_MTA_IP]:PORT): 250 2.0.0 Ok: queued as TheNextQueueID)", "postfix_queueid" : "TheQueueID", "postfix_message-id" : null, "tags" : [ "_grok_postfix_success" ] } }, { "_index" : "My_Elasticsearch_Index", "_type" : "_doc", "_id" : "The_Document_2_ID", "_score" : 1.0, "_source" : { "path" : "the/path/to/the/log/filter", "subject" : null, "postfix_to" : "recipient@domain.tld", "postfix_status" : "sent", "timestamp" : "Nov 2 02:30:22",, "postfix_from" : "sender@domain.tld", "@timestamp" : "2021-11-02T02:30:22.656Z", "program" : "postfix/smtp", "message" : "TheQueueID: to=, relay=The_Relay_Hostname[The_Relay_IP]:PORT, delay=0.15, delays=0.01/0.03/0/0.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[The_MTA_IP]:PORT): 250 2.0.0 Ok: queued as TheNextQueueID)", "postfix_queueid" : "TheQueueID", "postfix_message-id" : "Some_Random_Numbers_and_Word_that_define_the_message_ID@domain.tld",, "tags" : [ "_grok_postfix_success" ] } }, ``` The first output has a subject but no message id, the second output has a message id but no subject. So subject and message id cannot be in the same document. What I want is the subject and message id in the same document. Is there any way to solve my problem? Please for the help. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 4, 2021, 5:38pm UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397/2 "2021-11-04T17:38:06Z")
You could do something like this... ``` mutate { gsub => ["message", "\s+", " "] } dissect { mapping => { "message" => "%{[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} %{} %{program}[%{}]: %{[@metadata][restOfLine]}" } } date { match => ["[@metadata][ts]", "MMM dd HH:mm:ss" ] } grok { break_on_match => false match => { "[@metadata][restOfLine]" => [ "^(?[0-9A-Z]{10}):", "Subject: (?[^;]*);", " from=(?[^,]*),", " message-id=%{GREEDYDATA:message_id}", " status=%{WORD:status}", " to=(?[^,]*)," ] } } if [taskId] { aggregate { task_id => "%{taskId}" code => ' map["@timestamp"] ||= event.get("@timestamp") map["message-id"] ||= event.get("message_id") map["postfixTo"] ||= event.get("postfixTo") map["postfixFrom"] ||= event.get("postfixFrom") map["subject"] ||= event.get("subject") ' push_map_as_event_on_timeout => true timeout_task_id_field => "taskId" timeout => 10 } } ``` This is just an outline of an approach, it omits quite a few fields. ---
**Author:** ![RichYaNa](https://avatars.discourse-cdn.com/v4/letter/r/ba8739/32.png) [@RichYaNa](https://discuss.elastic.co/u/RichYaNa)\ **Post date:** [November 5, 2021, 3:07am UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397/3 "2021-11-05T03:07:29Z")
Hi Badger, Thank you for your answer, where exactly should I put the filter? After the 31 Filter? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 5, 2021, 3:12am UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397/4 "2021-11-05T03:12:54Z")
> [@RichYaNa](#): > > where exactly should I put the filter? It is not a drop-in addition to your existing configuration, not is it a complete configuration by itself. You may not need anything from it except some ideas from the aggregate, and that will need to be updated to adjust field names. As I said, it is just an outline of an approach. ---
**Author:** ![RichYaNa](https://avatars.discourse-cdn.com/v4/letter/r/ba8739/32.png) [@RichYaNa](https://discuss.elastic.co/u/RichYaNa)\ **Post date:** [November 15, 2021, 4:29am UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397/5 "2021-11-15T04:29:43Z")
Hi Badger, Can you explain to me what this symbol means in the aggregation filter code section: > [@Badger](#): > > `||=` ```auto ||= = == ``` or is there a reference explaining my question that you can share with me? thanks. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 15, 2021, 4:42am UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397/6 "2021-11-15T04:42:33Z")
> [@Badger](#): > > ||= That is a conditional assignment. I'm using it to say > If map["postfixTo"] is nil, then set it to the value of event.get("postfixTo") (which may itself be nil). Basically, it sets the map value to the content of the field from the first event that contains it (that's usually what I want for something like @timestamp). There is more to it. See [here](https://riptutorial.com/ruby/example/10905/or-equals-conditional-assignment-operator------) for one explanation. You should be able to Google many more pages once you have the wording from that one. ---
**Author:** ![RichYaNa](https://avatars.discourse-cdn.com/v4/letter/r/ba8739/32.png) [@RichYaNa](https://discuss.elastic.co/u/RichYaNa)\ **Post date:** [December 6, 2021, 9:48am UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397/7 "2021-12-06T09:48:12Z")
Thank you for the help @Badger its help me so much. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 3, 2022, 9:48am UTC](https://discuss.elastic.co/t/try-to-parsing-postfix-log-using-logstash/288397/8 "2022-01-03T09:48:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.