Hello,
I'm trying to parse Exim logs which spread through multiple lines
Here is example of logs:
2017-03-28 12:26:49 1cswlV-0004KG-2c DKIM: d=gmail.com s=20161025 c=relaxed/relaxed a=rsa-sha256 b=2048 [verification succeeded]
2017-03-28 12:26:55 1cswlV-0004KG-2c <= test@gmail.com H=mail-oi0-f54.google.com [209.85.218.54] P=esmtps X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no S=3701 id=CAPS5ocfZJhwpH9WtWNNAug
EBWY9QBz9gbijQUQFoYK71OwrbTQ@mail.gmail.com T="test subject"
2017-03-28 12:26:55 1cswlV-0004KG-2c => test user test@testserver.com R=mysql_user T=mysql_delivery
2017-03-28 12:26:55 1cswlV-0004KG-2c Completed
Originally I parse it line by line and that works ok but it hard to find specific email info with single query since the only common field is 1cswlV-0004KG-2c across all lines.
I recently found that you can use the codec to add rows matching the same pattern:
input {
if [type] == "exim-log" {
beats {
port => "6001"
codec => multiline {
# Grok pattern names are valid! :)
pattern => "^%{TIMESTAMP_ISO8601} %{EXIM_MSGID} "
negate => false
what => previous
}
}
}
Worked fine if there were only one message in same time, but if there were other transactions at the same time they were added to message value as well.
So here is example of my message value if server accepts two in same time:
2017-03-28 12:47:11 1csx5C-00050B-Tz DKIM: d=gmail.com s=20161025 c=relaxed/relaxed a=rsa-sha256 b=2048 [verification succeeded]
2017-03-28 12:47:17 1csx5C-00050B-Tz <= test@gmail.com H=mail-oi0-f41.google.com [209.85.218.41] P=esmtps X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no S=3610 id=CAPS5occ8Hs-N5LFygHOzNf=Yp9pv0M1BRtvb_13mQzhoOw@mail.gmail.com T="test 2"
2017-03-28 12:47:17 1csx5C-00050B-Tz => test test@yahoo.com R=mysql_user T=mysql_delivery
2017-03-28 12:47:17 1csx5C-00050B-Tz Completed
2017-03-28 12:47:43 1csx5j-00051K-A7 DKIM: d=gmail.com s=20161025 c=relaxed/relaxed a=rsa-sha256 b=2048 [verification succeeded]
2017-03-28 12:47:46 1csx5j-00051K-A7 H=mail-oi0-f41.google.com [209.85.218.41] X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no F=test@gmail.com rejected after DATA: This message is rejected by the Anti-Spam System. Spam-score too high : 1002.9 spam points - Please reformat your email and send again
In other words instead of creating separate id for 1csx5C-00050B-Tz and 1csx5j-00051K-A7 it combine it together
I'm sending these logs from other server using filebeats.
Please advise how to properly combine these records.