Hi,
My grok keeps timing out while trying to filter a not so complicated/long log file. I have been trying for days now to no avail. I have tested all the patterns in the grok constructor with no issue so the main problem seems to be how the pattern is reading the log to cause a timeout, and not the integrity of the patterns. However, it is possible that my patterns aren't optimal because I am new at this.
The sample log file I am trying to filter is:
Error Code: E-00000
Severity: WORDS
Category: CATEGORYTYPE.CATEGORY
Timestamp: 2016-09-08 06:08:12.621
Message: MESSAGE ERROR
Exception: TextTextTextTextTextTextTextText TextTextTextTextTextTextTextTextTextTextTextText
TextTextTextTextText
at TxtTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTe
at xtTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTex
at TextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTextTe
Machine: XX00XXXX000X
Application Domain: Application.exe
Process Id: 0000
Process Name: C:\file\file\filehere\file.exe
App Type for Log: NAME
Win32 Thread Id: 0000
Thread Name:
Extended Properties:
Where my grok looks like this, and all the 3 letter patterns are just the names of each category and won't be displayed. The problem seems to be from the filter because when I take out the "Exception: " field from the log and filter, everything runs perfectly smoothly.:
filter{
grok {
timeout_millis => 60000
patterns_dir => "/etc/logstash/patterns/patterns"
match => { "message" => "%{ERC}%{CODE:ErrorCode456}%{SPACE}%{SEV}%{WORD:Severity}%{SPACE}%{CAT}%{GREEDYDATA:Category}%{SPACE}
%{TIM}%{TIMESTAMP_ISO8601:Time}%{SPACE}%{MES}(?<Summary>%{SPACE}(?:(?!((Machine: )|(Exception: ))).)*)%{SPACE}%{EXC}(?<Exception>(.*\s*)*(?:(?!Machine: ).)*)
%{SPACE}%{MACH}%{SERVER:Server}%{SPACE}%{APD}(?<ApplicationDomain>.*exe$)%{SPACE}%{PID}%{INT:PID}%{SPACE}%{PRN}(?<Process Name>.*exe$)%{SPACE}%{ATL}
%{APPTYPE:App Type for Log}%{SPACE}%{WTI}%{NUMBER:ThreadID}%{SPACE}%{THN}%{SPACE}%{EXP}" }
}
}
And my patterns are:
ERC Error Code:\s*
CODE \w-\d{5}
SEV Severity:\s*
CAT Category:\s*
TIM Timestamp:\s*
EXC Exception:\s*
MACH Machine:\s*
SERVER \w+{8,}
APD Application Domain:\s*
PID Process Id:\s*
PRN Process Name:\s*
ATL App Type for Log:\s*
APPTYPE \w*
WTI Win32 Thread Id:\s*
THN Thread Name:\s*
EXP Extended Properties:\s*
This is the error I get in case that it's of any help.
[2016-11-11T15:21:06,063][WARN ][logstash.filters.grok ] Timeout executing grok '%{ERC}%{CODE:ErrorCode456}%{SPACE}%{SEV}%{WORD:Severity}%{SPACE}%{CAT}%{GREEDYDATA:Category}%{SPACE}%{TIM}%{TIMESTAMP_ISO8601:Time}%{SPACE}%{MES}(?<Summary>%{SPACE}(?:(?!((Machine: )|(Exception: ))).)*)%{SPACE}%{EXC}(?<Exception>(.*\s*)*(?:(?!Machine: ).)*)%{SPACE}%{MACH}%{SERVER:Server}%{SPACE}%{APD}(?<ApplicationDomain>.*exe$)%{SPACE}%{PID}%{INT:PID}%{SPACE}%{PRN}(?<Process Name>.*exe$)%{SPACE}%{ATL}%{APPTYPE:App Type for Log}%{SPACE}%{WTI}%{NUMBER:ThreadID}%{SPACE}%{THN}%{SPACE}%{EXP}' against field 'message' with value 'Value too large to output (791 bytes)! First 255 chars are: