Help with grok patterns for custom Postfix logs

[decibel83]

Hi,

I am parsing Postfix logs using the good work from https://github.com/topmedia/logstash-postfix.
Now I'm trying to customise the grok patterns because I have to parse some custom logs created with the Postfix header_checks feature (example at https://serverfault.com/questions/392838/how-to-log-extension-header-in-postfix-log).

The use case is to parse and import custom headers name and value which are written in Postfix logs like these:

Oct 24 15:18:09 server postfix/cleanup[30021]: 7D5F140AAA: warning: header X-Header-Custom1: Bla bla bla from unknown[123.123.123.123]; from=<user@domain.com> to=<user@domain.com> proto=ESMTP helo=<client.domain.com>
Oct 24 15:18:09 server postfix/cleanup[30021]: 7D5F140AAA: warning: header X-Header-Custom2: Test test test from unknown[123.123.123.123]; from=<user@domain.com> to=<user@domain.com> proto=ESMTP helo=<client.domain.com>
Oct 24 15:18:09 server postfix/cleanup[30021]: 7D5F140AAA: warning: header Subject: Mail test with custom headeer from unknown[123.123.123.123]; from=<user@domain.com> to=<user@domain.com> proto=ESMTP helo=<client.domain.com>

These logs are parsed with the following grok patterns (you can see full patterns at https://github.com/topmedia/logstash-postfix/blob/master/etc/logstash/patterns.d/postfix.grok):

common postfix patterns
POSTFIX_QUEUEID ([0-9A-F]{6,}|[0-9a-zA-Z]{12,})
POSTFIX_WARNING_LEVEL (warning|fatal|info)
POSTFIX_KEYVALUE_DATA [\w-]+=[^;]*

# warning patterns
POSTFIX_WARNING_WITH_KV (%{POSTFIX_QUEUEID:postfix_queueid}: )?%{POSTFIX_WARNING_LEVEL:postfix_message_level}: %{GREEDYDATA:postfix_message}; %{POSTFIX_KEYVALUE_DATA:postfix_keyvalue_data}
POSTFIX_WARNING_WITHOUT_KV (%{POSTFIX_QUEUEID:postfix_queueid}: )?%{POSTFIX_WARNING_LEVEL:postfix_message_level}: %{GREEDYDATA:postfix_message}
POSTFIX_WARNING %{POSTFIX_WARNING_WITH_KV}|%{POSTFIX_WARNING_WITHOUT_KV}

With this patterns I get the following result on the first log (I'm debugging at http://grokdebug.herokuapp.com/):

{
  "POSTFIX_WARNING_WITHOUT_KV": [
    [
      null
    ]
  ],
  "postfix_queueid": [
    [
      null,
      "7D5F140AAA"
    ]
  ],
  "postfix_message_level": [
    [
      null,
      "warning"
    ]
  ],
  "postfix_message": [
    [
      null,
      "header X-Header-Custom1: Bla bla bla from unknown[123.123.123.123]"
    ]
  ],
  "POSTFIX_WARNING_WITH_KV": [
    [
      "7D5F140AAA: warning: header X-Header-Custom1: Bla bla bla from unknown[123.123.123.123]; from=<user@domain.com> to=<user@domain.com> proto=ESMTP helo=<client.domain.com>"
    ]
  ],
  "postfix_keyvalue_data": [
    [
      "from=<user@domain.com> to=<user@domain.com> proto=ESMTP helo=<client.domain.com>"
    ]
  ]
}

What I want to get is two new metadata:

• postfix_header_name: X-Header-Custom1
• postfix_header_value: Bla bla bla

I tried to add two new custom patterns:

POSTFIX_HEADER_NAME Subject|X-[A-z\-]+
POSTFIX_HEADER_VALUE .+

and tried to change the POSTFIX_WARNING_WITH_KV pattern with the following:

POSTFIX_WARNING_WITH_KV (%{POSTFIX_QUEUEID:postfix_queueid}: )?%{POSTFIX_WARNING_LEVEL:postfix_message_level}: (header %{POSTFIX_HEADER_NAME:postfix_header_name}: {POSTFIX_HEADER_VALUE:postfix_header_value} from)? %{GREEDYDATA:postfix_message}; %{POSTFIX_KEYVALUE_DATA:postfix_keyvalue_data}}

but this don't work.

Could you help me to understand where I'm wrong, please?