Parse Exim logs

I'm trying to parse Exim logs which spread through multiple lines
Here is example of logs:

2017-03-28 12:26:49 1cswlV-0004KG-2c DKIM: s=20161025 c=relaxed/relaxed a=rsa-sha256 b=2048 [verification succeeded]
2017-03-28 12:26:55 1cswlV-0004KG-2c <= [] P=esmtps X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no S=3701 id=CAPS5ocfZJhwpH9WtWNNAug T="test subject"
2017-03-28 12:26:55 1cswlV-0004KG-2c => test user R=mysql_user T=mysql_delivery
2017-03-28 12:26:55 1cswlV-0004KG-2c Completed

Originally I parse it line by line and that works ok but it hard to find specific email info with single query since the only common field is 1cswlV-0004KG-2c across all lines.

I recently found that you can use the codec to add rows matching the same pattern:

input {
if [type] == "exim-log" {

beats {
    port => "6001"

    codec => multiline {
    # Grok pattern names are valid! :)
    pattern => "^%{TIMESTAMP_ISO8601} %{EXIM_MSGID} "
    negate => false
    what => previous


Worked fine if there were only one message in same time, but if there were other transactions at the same time they were added to message value as well.
So here is example of my message value if server accepts two in same time:

2017-03-28 12:47:11 1csx5C-00050B-Tz DKIM: s=20161025 c=relaxed/relaxed a=rsa-sha256 b=2048 [verification succeeded]
2017-03-28 12:47:17 1csx5C-00050B-Tz <= [] P=esmtps X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no S=3610 T="test 2"
2017-03-28 12:47:17 1csx5C-00050B-Tz => test R=mysql_user T=mysql_delivery
2017-03-28 12:47:17 1csx5C-00050B-Tz Completed
2017-03-28 12:47:43 1csx5j-00051K-A7 DKIM: s=20161025 c=relaxed/relaxed a=rsa-sha256 b=2048 [verification succeeded]
2017-03-28 12:47:46 1csx5j-00051K-A7 [] X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no rejected after DATA: This message is rejected by the Anti-Spam System. Spam-score too high : 1002.9 spam points - Please reformat your email and send again

In other words instead of creating separate id for 1csx5C-00050B-Tz and 1csx5j-00051K-A7 it combine it together

I'm sending these logs from other server using filebeats.
Please advise how to properly combine these records.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.