How to grok different format record lines in one single log file

Elastic Stack Logstash
1

Dear Friend,

I'm encountering an issue when configuring logstash, here is an example from my case that I need to grok filter to match - some lines are with IP fields and sesseionID fields but some are not (so they are not exacly the same fields in one single log file). How can I modify the grok expression to cater for this?

"SMTPD" 1882 4499 "2020-06-11 18:18:18.188" "192.168.8.98" "SENT:220 MAILSER ESMTP"
"SMTPC" 1572 4898 "2020-06-11 18:18:18.925" "192.168.8.97" "SENT: 250 Queued"
"APPLICATION" 1767 "2020-06-11 18:18:18.925" "SMTPDeliverer - Message 3458327"
"TCP" 1992 "2020-06-11 18:18:18.925" "TCP - 192.168.8.98 connect to 192.168.8.99"
"TCPIP" 1992 "2020-06-11 18:18:18.925" "TCP - 192.168.8.98 connect to 192.168.8.99"

All lines start with quotation mark ", that's something annoying. You can see SMTP* are 2 more fields than APPLICATION, TCP lines. I tried to use two message expression in the same grok:

["message", ''"SMTP*""\s*(%{BASE10NUM:process})\s*(%{BASE10NUM:session})\s*...],
["message", ''"APP*""\s*(%{BASE10NUM:process})\s*...]

But it didn't work this way. Any help or advice would be greatly appreciated!

2

Two grok matches should work:
grok{
match => ["message", ''"SMTP*""\s*(%{BASE10NUM:process})\s*(%{BASE10NUM:session})\s*...]
match => ["message", ''"APP*""\s*(%{BASE10NUM:process})\s*...]
}

3

Badly that it didn't work.

Firstly I think we need to use " instead of ", but it's still not working by doing so. I tried it with Grij Debugger by using "SMTP*" but it didn't match any patterns as listed. Any ideas?

How we can let grok to match the first expression when it's starting with "SMPT
then grok to match the second expression when the line of log starting with "APP*

4

Dear Wang, I found a way out (looks like that):

I have to match exactly SMTPD for the first match expression by using "SMTPD", then use "\APPLICATION" for the other expression. it works this way! but if I got too many types of works to start, I'm trying to figure out how to use the first one to start with "SMTP and the other express will match those lines not starting with "SMTP

5

You can make fields optional by surrounding them with ( )? So this will parse those messages

grok { match => { "message" => '"%{WORD:app}" %{INT:number1}( %{INT:number2})? %{QS:string1} %{QS:string2}( %{QS:string3})?' } }
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.