I have multiple files each containing a single email in the following format:
Message-ID: <18435268.1075855378308.JavaMail.evans@thyme>
Date: Mon, 7 May 2001 12:28:00 -0700 (PDT)
From: phillip@xxx.com
To: matthew@xxx.com, jay@xxx.com, matt@xxx.com
Subject: fun and games
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-From: Phillip K
X-To:
X-cc:
X-bcc:
X-Folder: \Phillip-Jan2002_1\Sent Mail
X-Origin: Philip
X-FileName: phillip (Non-Privileged).pst
[1]
[2]some text
[3]some more text
[4]
[5]
[6]---
[7]
[8]😎😎
[9]
[10]----
[11]
[12] even more text
I have created the following Grok patterns:
Message-ID: <%{NOTSPACE:messageID}>
Date: %{DATA:dateString}
From: %{NOTSPACE:from}
To: %{DATA:to}
Subject: (?<subject>(?:(.*)|\n|\r\n?)((?:(?:\n|\r\n?).*)*))
Mime-Version: %{NOTSPACE:mimeVersion}
Content-Type: %{DATA:contentType}
Content-Transfer-Encoding: %{NOTSPACE:contentTypeTransferEncoding}
X-From: %{DATA:xFrom}
X-To: %{DATA:xTo}
X-cc: %{DATA:xCC}
X-bcc: %{DATA:xBCC}
X-Folder: %{DATA:xFolder}
X-Origin: %{DATA:xOrigin}
X-FileName: %{DATA:xFilename}
(?<messageBody>(?:(.*)|\n|\r\n?)((?:(?:\n|\r\n?).*)*))
which all together successfully parses the above email example using the 'Grok Debugger without any issues.
Now being new to logstash (and elastic search) I need some help to convert this pattern into a logstash config file that will parse each email file and create a single event with this data (and then ingest as JSON document etc)
I have tried the following but no joy -- it doesn't out seem to output the lines nor give me any errors with --verbose or --debug options on
input {
stdin {
codec => multiline {
pattern => "(^Message-ID: <%{NOTSPACE:messageID}>)|(^Date: %{DATA:dateString})|(^From: %{NOTSPACE:from})|(^To: %{DATA:to})|(^Subject: (?<subject>(?:(.*)|\n|\r\n?)((?:(?:\n|\r\n?).*)*)))|(^Mime-Version: %{NOTSPACE:mimeVersion})|(^Content-Type: %{DATA:contentType})|(^Content-Transfer-Encoding: %{NOTSPACE:contentTypeTransferEncoding})|(^X-From: %{DATA:xFrom})|(^X-To: %{DATA:xTo})|(^X-cc: %{DATA:xCC})|(^X-bcc: %{DATA:xBCC})|(^X-Folder: %{DATA:xFolder})|(^X-Origin: %{DATA:xOrigin})|(^X-FileName: %{DATA:xFilename})|((?<messageBody>(?:(.*)|\n|\r\n?)((?:(?:\n|\r\n?).*)*)))"
negate => true
what => "previous"
}
}
}
output {
stdout { }
stdout { codec => rubydebug }
}
Any help would be appreciated -- obviously I am missing something here ....
Try this, with a literal newline embedded in the string