GROKPARSE Failure and after Successful debug

Elastic Stack Logstash
1

Hi, i have used https://grokdebug.herokuapp.com/ to create grok match patterns and which are worked fine. However when checking with logstash, some of fields are not extracted and getting grokparse failure error.

Sample log

####<Sep 9, 2020 11:13:29,426 AM EDT> <[ACTIVE] ExecuteThread: '318' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <4325b15b-61b9-47b6-b7ce-3931a3c348bc-00056b95> <1599664409426> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <[ServletContext@1011161545[app:SPLWeb module:/ouaf path:null spec-version:3.1]] Root cause of ServletException.
java.lang.IndexOutOfBoundsException: No group 1
at java.util.regex.Matcher.start(Matcher.java:375)
at java.util.regex.Matcher.appendReplacement(Matcher.java:880)
at java.util.regex.Matcher.replaceAll(Matcher.java:955)
at java.lang.String.replaceAll(String.java:2223)
at com.splwg.base.web.common.ServletHelper.getErrorResponse(ServletHelper.java:1004)
at com.splwg.base.web.userMap.AbstractUiMapServlet.doPost(AbstractUiMapServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3667)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2443)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2291)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2269)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1703)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1663)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)

####<Sep 9, 2020 11:13:29,434 AM EDT> <[ACTIVE] ExecuteThread: '99' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <4325b15b-61b9-47b6-b7ce-3931a3c348bc-00056b98> <1599664409434> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <Watch "UncheckedException" in module "Module-FMWDFW" with severity "Notice" on server "Jvmname" has triggered at Sep 9, 2020 11:13:29 AM EDT. Notification details:
WatchRuleType: Log
WatchRule: (log.severityString == 'Error') and ((log.messageId == 'WL-101020') or (log.messageId == 'WL-101017') or (log.messageId == 'WL-000802') or (log.messageId == 'BEA-101020') or (log.messageId == 'BEA-101017') or (log.messageId == 'BEA-000802'))
WatchData: MESSAGE = [ServletContext@1011161545[app:SPLWeb module:/ouaf path:null spec-version:3.1]] Root cause of ServletException.
java.lang.IndexOutOfBoundsException: No group 1
at java.util.regex.Matcher.start(Matcher.java:375)
at java.util.regex.Matcher.appendReplacement(Matcher.java:880)
at java.util.regex.Matcher.replaceAll(Matcher.java:955)
at java.lang.String.replaceAll(String.java:2223)
at com.splwg.base.web.common.ServletHelper.getErrorResponse(ServletHelper.java:1004)
at com.splwg.base.web.userMap.AbstractUiMapServlet.doPost(AbstractUiMapServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.splwg.base.web.services.RequestContextFilter.doFilter(RequestContextFilter.java:64)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.splwg.base.web.services.HeaderSecurityFilter.doFilter(HeaderSecurityFilter.java:41)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.splwg.base.web.utility.OJETMappingFilter.doFilter(OJETMappingFilter.java:76)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.splwg.base.web.utility.DoctypeReplaceFilter.doFilter(DoctypeReplaceFilter.java:44)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.splwg.base.web.services.SecurityFilter.forwardRequest(SecurityFilter.java:91)
at com.splwg.base.web.services.SecurityFilter.doFilter(SecurityFilter.java:45)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.splwg.base.web.utility.SessionTimeOutFilter.doFilter(SessionTimeOutFilter.java:49)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.splwg.base.web.utility.CompressionFilter.doFilter(CompressionFilter.java:46)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at oracle.security.jps.ee.http.JpsAbsFilter$3.run(JpsAbsFilter.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:650)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:110)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilterInternal(JpsAbsFilter.java:273)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:147)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:94)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:248)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3701)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3667)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2443)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2291)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2269)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1703)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1663)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
SUPP_ATTRS = {severity-value=8, rid=0, partition-id=0, partition-name=DOMAIN} SERVER = ccbsprd_ms01 TIMESTAMP = 1599664409426 USERID = MACHINE = SJI-CCBSJG-AP01 MSGID = BEA-101017 DATE = Sep 9, 2020 11:13:29,426 AM EDT SUBSYSTEM = HTTP CONTEXTID = 4325b15b-61b9-47b6-b7ce-3931a3c348bc-00056b95 TXID = THREAD = [ACTIVE] ExecuteThread: '318' for queue: 'weblogic.kernel.Default (self-tuning)' SEVERITY = Error
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 30000

logstash.conf

input {
file {
path => "/var/log/.log"
start_position => "beginning"
type => "log"
}
}
filter {
if [type] == "log" {
multiline {
pattern => "^(?!####)"
negate => false
what => "previous"
}
grok {
match => [
"message", "####<(?%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (?:AM|am|PM|pm)) %{TZ}>%{SPACE}<%{LOGLEVEL:severity}>%{SPACE}<%{WORD:loginfo}>%{SPACE}<%{USERNAME:servername}>%{SPACE}<%{USERNAME:jvmname}>%{SPACE}<((?(.?))>)%{SPACE}<((?(.?))>)%{SPACE}<((?(.?))>)%{SPACE}<((?(.?))>)%{SPACE}<((?(.?))>)%{SPACE}<((?(.?))>)%{SPACE}<((?(.?))>)%{SPACE}<((?(.?))>)",
"message", "\A####<(?%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (?:AM|am|PM|pm)) %{TZ}> <%{LOGLEVEL:severity}> <%{WORD:loginfo}> <%{USERNAME:servername}> <%{USERNAME:jvmname}> <((?(.?))>) <((?(.?))>) <((?(.?))>)%{SPACE}<((?(.?))>) <((?(.?))>) <((?(.?))>) <((?(.?))>) <%{DATA:ccbsrequiredmessage}({({[^}]+},?\s)})?\s$(?(?m:.*))?>"]
}

if [ccbsrequiredmessage] =~ "the total memory in the server" {
mutate {
split => ["ccbsrequiredmessage"," "]
add_field => { "AvailableMemory" => "%{[ccbsrequiredmessage][0]}" }
}
mutate {
gsub => ["AvailableMemory","%",""]
}
mutate {
add_field => {"TotalMemory" => "100"}
}
mutate {
convert => {"AvailableMemory" => "integer"}
convert => {"TotalMemory" => "integer"}
}
ruby {
code => "event.set('UsedMemory',event.get('TotalMemory')-event.get('AvailableMemory'))"
}
}
if [ccbsrequiredmessage] =~ "thread pool contains" {
mutate {
split => ["ccbsrequiredmessage"," "]
add_field => { "RunningThreads" => "%{[ccbsrequiredmessage][4]}" }
add_field => { "IdleThreads" => "%{[ccbsrequiredmessage][7]}" }
add_field => { "StandbyThreads" => "%{[ccbsrequiredmessage][11]}" }
}
}
if [ccbsrequiredmessage] =~ "Size based data retirement operation completed" {
mutate {
split => ["ccbsrequiredmessage"," "]
add_field => { "RetiredRecords" => "%{[ccbsrequiredmessage][10]}" }
add_field => { "Retire_ResponseTime" => "%{[ccbsrequiredmessage][13]}" }
add_field => { "Archive_details" => "%{[ccbsrequiredmessage][8]}" }
}
}
mutate {
convert => { "RunningThreads" => "integer"}
convert => { "IdleThreads" => "integer"}
convert => { "StandbyThreads" => "integer"}
convert => { "RetiredRecords" => "integer"}
convert => { "Retire_ResponseTime" => "float"}
# convert => { "AvailableMemory" => "integer"}
convert => { "UsedMemory" => "integer"}
}
date {
match => [ "timestamp7", "MMM dd, yyyy hh:mm:ss,SSS a" ]
timezone => "Asia/Kolkata"
target => "@timestamp"
}
}
}
output {
stdout { codec => rubydebug }
}

Not able to extract stack trace field and for some of lines of data the strack trace data is coming into ccbsrequiredmessage field.

But which is working fine grok debugger and grok constructor sites.

Please help me how to fix this issue.

2

I never use grokdebug.herokuapp.com or other grok debuggers (including kibana) because they sometimes interpret ambiguous regexps differently to grok. And many regexps that use DATA, or especially GREEDYDATA, are ambiguous. The process I suggest to develop complex grok patterns is documented here.

Please edit your post, select the log file entries and click on </> in the toolbar above the edit pane. In the preview pane you will see the display change to

at java.util.regex.Matcher.appendReplacement(Matcher.java:880)
at java.util.regex.Matcher.replaceAll(Matcher.java:955)
at java.lang.String.replaceAll(String.java:2223)

Then do the same for the configuration. Please do not do it to the whole post.

3

Hi Badger,

Can you please help for capturing of stack trace of one log event. Rest of other fields I will taken care.

I tried the options provided in link but it is not working in my case.
might be I have written wrong.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.