Grok for email log

(Vikash Singh) #1

Can you please help me with building a grok pattern for SMTP log:
Example of my smtp log:

Received: from
by n-mail-1.xxxxx.inwith LMTP id uOV8CFLmalxdRQAADPfE8A
for; Mon, 18 Feb 2019 22:37:30 +0530
Received: from ( [])
by (Postfix) with ESMTP id 1B245263926;
Mon, 18 Feb 2019 22:37:30 +0530 (IST)
Received: from (localhost [])
by (Postfix) with ESMTP id 04FE12DA1D2;
Mon, 18 Feb 2019 22:37:30 +0530 (IST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 1.163
X-Spam-Level: *
X-Spam-Status: No, score=1.163 tagged_above=-999 required=6
autolearn=no autolearn_force=no
Received: from ([])
by ( []) (amavisd-new, port 10024)
with ESMTP id bYziy9Hx1yoW; Mon, 18 Feb 2019 22:37:29 +0530 (IST)
Received: from localhost (localhost [])
by (Postfix) with ESMTP id 900F22D6BD6;
Mon, 18 Feb 2019 22:37:29 +0530 (IST)
Date: Mon, 18 Feb 19 17:05:17 +0000
Subject: Mail from Trade Apprenticeship Scheme at xxxxx (TASAR-2019)

Dear Candidate,

Your password for Online Application Submission for the Apprenticeship Program of trade Electrician against
Advertisement No.xxxxxxxx and trade code I-8 is japupu3ys

Login name is same as your Email ID.

This is system generated mail, please do not reply to it.


For each header that you want to extract use a pattern that matches zero-or-more characters that are not newline followed by a newline.

    grok { match => { message => "^Header: (?<Header>[^
" } }

Then you can give the grok filter an array of patterns to match.

(Vikash Singh) #3

Please elaborate little bit more.


I assume that you consume the entire log as a single event using multiline codec. If you are consuming it one line at a time things will be a little different. Decide which headers you want to parse and use the pattern I described to extract them. For example

    grok {
        break_on_match => false
        match => {
            message => [
                "^Date: (?<Date>[^
                "^Subject: (?<Subject>[^
                "^X-Spam-Level: (?<X-Spam-Level>[^
                "^X-Spam-Status: (?<X-Spam-Status>[^

Once you have extracted them you may require additional parsing to extract fields from those headers.

For Received you cannot do it using grok because there are multiple occurences, however, it is simple to do a similar regex match in ruby

    ruby {
        code => 'event.set("Received", event.get("message").scan(/^Received: ([^

That will set Received to an array of 5 strings which contain the contents of the Received headers.

(Vikash Singh) #5

actually I am newborn for elk and I am still unable to understand how to use multiline filter


How are you ingesting the log?

(Vikash Singh) #7

via FileBeat


Do you want to ingest the entire file as a single event?

(Vikash Singh) #9



If filebeat works the same way as a multiline codec, then you should be able to do that by specifying a pattern that never matches

multiline.pattern: '^Spalanzani'
multiline.negate: true
multiline.match: after
(Vikash Singh) #11

Is there any other way to segregate multi lines logs other than groks?


You can use other filters. What are you trying to do and what problem are you having using grok?

(Vikash Singh) #13

Other filters!!! Like? I am not able define grok

(Vikash Singh) #14

Since the mail log is unstructured how can I define one common filter which would read all the logs correctly?

(system) closed #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.