How to parse single log file with multiple grok pattern

Elastic Stack Logstash
1

Hello,
I am trying to parse a single log file with multiple grok pattern as below:

grok {
break_on_match => false
match => {"message" => ["%{TIMESTAMP_ISO8601:timestamp_match}%{SPACE}[%{SPACE}%{WORD:number}]%{SPACE}[[^[]]]%{SPACE}[%{SPACE}%{WORD:demo_no}]%{SPACE}[%{WORD:log_level}]%{SPACE}[%{WORD:info}]", "%{TIMESTAMP_ISO8601:timestamp_match}%{SPACE}[%{SPACE}%{WORD:number}]%{SPACE}[[^[]]]%{SPACE}[%{SPACE}%{WORD:demo_no}]%{SPACE}[%{WORD:log_level}]%{SPACE}[%{WORD:soap_type}]%{SPACE}[%{WORD:info}]"
]
}

MY logs look like below:

+++ [AAANG] +++++++++++++++++++++++++++++++++++++++++
2017-04-26 07:59:44,884 [ 3] [bc9d7002-775f-40ed-8322-a2e3e5rrr66e] [ demo3] [DEBUG] [SoapRequest] [PW.Infrastructure.Web.Services.SoapLogExtension.WriteOutput]
2017-04-26 07:59:44,888 [ 3] [bc9d7002-775f-40ed-8322-a2e3e5rrr66e] [ demo3] [DEBUG] [<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">soap:BodyeeeWWWW</soap:Body></soap:Envelope>] [PW.Infrastructure.Web.Services.SoapLogExtension.WriteOutput]
2017-04-26 07:59:45,039 [ 3] [bc9d7002-775f-40ed-8322-a2e3e5rrr66e] [ demo3] [DEBUG] [SoapResponse] [PW.Infrastructure.Web.Services.SoapLogExtension.WriteInput]

Can u please point out what is wrong?

2

I don't exactly know, but I can at least tell you that if in your log you have a bracket, brace, etc., you need to escape it in your pattern:

[%{SPACE}%{WORD:number}]
shall become
\[%{SPACE}%{WORD:number}\]

And test your pattern with http://grokdebug.herokuapp.com/ for exemple.
If your pattern is correct, I think it might be OK (are you sure about break_on_match => false?)

3

I do not know why these back slashes disappeared while i posted this query but they look like below:

Grok Pattern.png1366×152 4.43 KB

I read about break_on_match from Grok Filter Reference document.

I need to write two patterns as the log file content are diferent for alternate line. :frowning:

4

If you're sure that both your patterns are correct, you can try to split them in two groks by doing:

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN1}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN2}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

If it does not work with this: your pattern is wrong somewhere
If it works: keep it or continue to look for the correct way to have multiple pattern in one grok

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.