Need logstash grok help

Elastic Stack Logstash
1

Hi,

My pattern works well in grok-debugger, but fails in Logstash's pipeline. I've been trying for hours; no luck. Here are the logs that I'm trying to parse:

  1. Nov 10 20:06:05 mx1 postfix/qmgr[4016]: D134620D03: from=<info@domain.com>, size=6638, nrcpt=1 (queue active)
  2. Nov 10 20:05:49 mx2 postfix/qmgr[4016]: 64FBB20D03: from=<>, size=5733, nrcpt=1 (queue active)
  3. Nov 10 20:06:05 mx1 postfix/qmgr[4016]: 38A6B2090E: removed

EIther of the filters work for all three log-patterns, in grok-debugger:

POST_QMGR %{SYSLOGBASE} %{QUEUEID:qid}: (?:removed|from=<(?:%{FROM_EMAIL:from})?>(?:, size=%{NUMBER:size}, nrcpt=%{NUMBER:nrcpt} \(%{DATA:queuestatus}\))?)

POST_QMGR %{SYSLOGBASE} %{QUEUEID:qid}: (removed|from=<(%{FROM_EMAIL:from})?>(, size=%{NUMBER:size}, nrcpt=%{NUMBER:nrcpt} \(%{DATA:queuestatus}\))?)

where:

FROM_EMAIL %{USERNAME:[from][emailname]}@%{HOSTNAME:[from][domain]}
QUEUEID (?:[A-F0-9]+|NOQUEUE)

But, neither of them work for log no.1, in the pipeline. What modifications should be done?

Thanks.

2

Please show your entire config file.

3

Please note that I've taken off %{SYSLOGBASE} from the aforementioned POST_QMGR syntax. So, now, my config file is:

input {
stdin {}
}

filter {

if [message] =~ /postfix\/qmgr\[\d+\]:/ {
grok {
patterns_dir => "/usr/share/logstash/patterns"
match => { "message" => "%{SYSLOGBASE} %{POST_QMGR}" }
tag_on_failure => [ "_grok_qmgr_fail" ]
add_tag => [ "_grok_postfix_success" ]
}
}

else {
mutate {
add_tag => ["_no_match"]
}
}

}

output {
elasticsearch {
hosts => [ "elk00.vpn:9200", "elk01.vpn:9200", "elk02.vpn:9200" ]
user => "logstash_logs"
password => "logstash_logs."
ssl => true
cacert => "/etc/logstash/ca.pem"
index => "logstash-postfix-%{+YYYY.MM.dd.HH.mm}"
}
stdout { codec => rubydebug }

This fails only for log no.1

Thanks.

4

I've found the pipeline failing (but successful in debugger) for another log and filter. The log-line is:

Nov 11 19:17:33 ezm06-pco postfix/pickup[7864]: 3364E62F1D: uid=5000 from=<maddistevens2001@gmail.com>

And the filter is:

POST_PICKUP %{QUEUEID:qid}: uid=%{NUMBER:uid} from=<(%{FROM_EMAIL:from})?>

The common thing here is the field from=<user@domain.tld> that is contributing to the failure. If it's just from=<>, the filter works well in pipeline.

Hope this helps.

5

Solved.

I replaced the initial syntax FROM_EMAIL with a simpler new syntax EMAIL:

EMAIL %{USERNAME:emailname}@%{HOSTNAME:domain}

It now works both in debugger and pipeline.

6

Thanks for sharing your solution!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.