If value in log string doesnot exist

Elastic Stack Logstash
#1

Hi all.
I have this log:

Oct 19 13:19:22 prod-mail amavis[2815]: (02815-17) Blocked SPAM {DiscardedInternal}, LOCAL [63.240.118.168]:41519 [63.240.118.168] some@mail.be -> some2@mail.ru, Queue-ID: 621CD1E07E7, Message-ID: 2E2B9DCEC5113C0FD3D69F220610E0C5@cld.be, mail_id: sRABc2HPCipB, Hits: 14.881, size: 53682, 2436 ms

And this config:

if [program] =~ "amavis" {
grok {
match => { "message" => "([0-9-]+)(?[A-z 0-9 -]+){[A-z]+}, %{GREEDYDATA}LOCAL (?[[]0-9.: a-f]+)<%{EMAIL:from}> -> <%{EMAIL:to}>,
Queue-ID: %{NOTSPACE:Queue-ID},( | Message-ID: <%{NOTSPACE:Message-ID}>, )mail_id: %{NOTSPACE:mail_id}, Hits: %{NUMBER:Hits}, %{GREEDYDATA:short_message}" }
overwrite => [ "short_message" ]
}
}

So if value in field "from" exists it parsed OK , but if in log we have nothing in field "from" such <> we get _grokparsefailure . How can resolved this problem ? Thx

#2

Use (|%{EMAIL:from}) instead of %{EMAIL:from}, i.e. allow either an empty string or an email address.

1 Like
#3

thank you again, Magnus
one more question ,

Oct 18 15:48:35 rshn-prod-mail amavis[26956]: (26956-12) Blocked INFECTED () {DiscardedInternal,Quarantined}, LOCAL [178.21.15.216]:44382 [178.21.15.216] <> -> some@mail.ru, quarantine: J/virus-JPrKMRLI_16V, Queue-ID: 688C11E0444, Message-ID: 491759eedfae671e149a18de207f7474a80f3416@17ros-sem.ru, mail_id: JPrKMRLI_16V, Hits: -, size: 1540239, 524 ms

this message doesnot parsed , so i add () in first part of config but it doesnot take effect.

config:

if [program] =~ "amavis" {
grok {
match => { "message" => "([0-9-]+)(?[A-z 0-9 - ()]+){[A-z]+}, %{GREEDYDATA}LOCAL (?[[]0-9.: a-f]+)<(|%{EMAIL:from})> -> <%{EMAIL:to}>,
Queue-ID: %{NOTSPACE:Queue-ID},( | Message-ID: <%{NOTSPACE:Message-ID}>, )mail_id: %{NOTSPACE:mail_id}, Hits: %{NUMBER:Hits}, %{GREEDYDATA:short_message}" }
overwrite => [ "short_message" ]
}
}

may be error with empty field Hits? In another messages it parsed well.

#4

Start with the simplest possible expression, e.g. ([0-9-]+) and make sure things work. Then add another token and check if it still works. Continue until you've found what part of the expression doesn't work.

#5