I'm pulling logs from a mail server called hmailserver into logstash and parsing them out with some of my own grok. I'm a complete noob at aggregating and despite reading every document on this site multiple times, I'm not sure how to use it. When mail is tagged as spam, spamassassin sends http code 554 like in the following logs.
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net 250 OK
"SMTPD" 2256 462887 "2018-02-26 13:48:31.839" "75.152.103.51" "SENT: 250 OK"
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net
"SMTPD" 2328 462887 "2018-02-26 13:48:32.003" "75.152.103.51" "RECEIVED: DATA"
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net 250 Hello.
"SMTPD" 2344 462887 "2018-02-26 13:48:31.749" "75.152.103.51" "SENT: 250 Hello."
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net
"SMTPD" 2256 462887 "2018-02-26 13:48:31.826" "75.152.103.51" "RECEIVED: MAIL FROM: ringcentral@banachtoomey.com"
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net 220 mx.mydomain.com
"SMTPD" 2256 462887 "2018-02-26 13:48:31.671" "75.152.103.51" "SENT: 220 mx.mydomain.com"
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net
"SMTPD" 2344 462887 "2018-02-26 13:48:31.749" "75.152.103.51" "RECEIVED: HELO banachtoomey.com"
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net
"SMTPD" 2208 462887 "2018-02-26 13:48:31.915" "75.152.103.51" "RECEIVED: RCPT TO: bigboss@mydomain.com"
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net 250 OK
"SMTPD" 2208 462887 "2018-02-26 13:48:31.917" "75.152.103.51" "SENT: 250 OK"
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net 354 OK, send.
"SMTPD" 2328 462887 "2018-02-26 13:48:32.005" "75.152.103.51" "SENT: 354 OK, send."
February 26th 2018, 13:48:33.965s75-152-103-51.ab.hsia.telus.net 554
"SMTPD" 2929 462887 "2018-02-26 13:48:32.609" "75.152.103.51" "SENT: 554 "
When I search for "554" in Kibana, I can see the number of spam messages blocked but that's all without drilling down. The message containing "554" has a "session" field to filter on that is identical across all the messages I posted above (one full convo between my server and another = one unique session). These logs come in one after another and often have other sessions interrupting and coming in between. How can I tell logstash "if you see http code 554, grab the session ID, search the session ID's for a "RCPT FROM:", list these as spam email addresses. Even aggregating all of those logs on session ID might not be so bad. I just wanted to ask you guys how to go about it. I'm a noob without a clue. Thanks