How to parse WebSphere SystemErr log in logstash

Elastic Stack Logstash
1

IBM WebSphere logs include SystemOut and SystemErr logs for a JVM. Each message in SystemOut is enabled with timestamp that can be parsed with logstash. Exceptions in this log has stack trace but does not include timestamp in front so that they can be combined into one message. Sample of SystemOut.log:
// Sample code
</>
[02.10.2020 12:00:00:742 TRT] 000000bc SystemOut O Fri Oct 02 12:00:00 TRT 2020 : PusulaQuartzJobListener.END : AsynchronousInvocation(pusulaQueueJob.asyncScan())
[02.10.2020 12:01:26:224 TRT] 0000005f NGUtil$Server I ASND0002I: Detected server alfa102 started on node valyriaNode01
[02.10.2020 12:04:04:835 TRT] 000001c0 filter E com.ibm.ws.webcontainer.filter.FilterInstanceWrapper doFilter SRVE8109W: Uncaught exception thrown by filter Seam Filter: java.io.FileNotFoundException: SRVE0190E: File not found: /favicon.ico
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:976)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:957)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:485)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:206)
at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)
at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:388)
at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:515)
at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:60)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at org.jboss.seam.web.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:42)
</>
Up to here parsing is fine. But when it comes to SystemErr.log each message has timestamp in front even if the stack trace has timestamp. Sample for SystemErr.log:
Starting of an error message:
</>
[18.09.2020 08:49:07:557 TRT] 0000016a SystemErr R org.mozilla.javascript.WrappedException: Wrapped java.lang.NullPointerException (Komut[700988]#5)
[18.09.2020 08:49:07:558 TRT] 0000016a SystemErr R at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:1754)
[18.09.2020 08:49:07:558 TRT] 0000016a SystemErr R at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:148)
[18.09.2020 08:49:07:558 TRT] 0000016a SystemErr R at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:225)
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.gen.Komut_700988__7._c_f_1(Komut[700988]:5)
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.gen.Komut_700988__7.call(Komut[700988])
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:74)
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.gen.Komut_700988__7._c_script_0(Komut[700988]:8)
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.gen.Komut_700988__7.call(Komut[700988])
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.gen.Komut_700988__7.call(Komut[700988])
[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R at org.mozilla.javascript.gen.Komut_700988__7.exec(Komut[700988])
...
[18.09.2020 08:49:07:567 TRT] 0000016a SystemErr R Caused by: java.lang.NullPointerException
[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R at com.gunessigorta.pusula.uretim.UretimScriptBean.endeksliDovizCinsiniAyarla(UretimScriptBean.java:6007)
[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R at java.lang.reflect.Method.invoke(Method.java:620)
[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R ... 112 more
[18.09.2020 08:49:07:577 TRT] 0000016a SystemErr R com.filika.sigorta.exceptions.IceAktarimException: Döviz cinsi kontrolü@RISK_BILGILERI[282456].end.fail, Teklif SIGORTA_KONULARI kuralları çalıştırılırken hata oluştu!
</>
How can I combine the same threadID lines as one message if the message does not start with "at"? I could not find how to write this rule in logstash. Pseudo could is like this
If the log is SystemErr
if threadID is same
if message is not starting by at
starting a new message as msgReal
else message starts with at
append the message to msgReal
else
set as new msgReal

should parse the SystemErr but not sure how to implement it in logstash. I appreciate your help. Thanks.

2

Hello Utku,

I am working on exactly same issue, do you have any good news to help me?

3

I appreciate if anyone helps me to parse the SystemErr.log of WebSphere with logstash.

4

Please edit your post, select the sections of log file, and click on </> in the toolbar above the edit pane. This change the formatting of the log files to be like this

[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R at java.lang.reflect.Method.invoke(Method.java:620)
[18.09.2020 08:49:07:568 TRT] 0000016a SystemErr R at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)

making it clear where the line breaks are.

5

I re-formatted the log sections.

6

I dropped both those log sections into a file and ingested it with a file input and this codec:

 codec => multiline {
      pattern => '^(\[\d{2}\.\d{2}\.\d{4} \d{2}:\d{2}:\d{2}:\d{3} \w{3}\] [0-9a-f]{8} SystemErr \w )?at' 
      negate => false 
      what => previous 
      auto_flush_interval => 1
 }

Basically build a pattern that matches "[18.09.2020 08:49:07:559 TRT] 0000016a SystemErr R " and make it optional by surrounding it with ( )?

If WebSphere interleaves lines from two different stack traces from two different threads you would have to use an aggregate filter.

7

My logstash file has this portion

filter {
  if ([fields][log_type] == "was") {
    grok {
          # was_shortname need to be regex, because numbers and $ can be in the word
          match => ["message", "\[%{DATA:wastimestamp} %{WORD:tz}\] %{BASE16NUM:was_threadID} (?<was_shortname>\b[A-Za-z0-9\$]{2,}\b) %{SPACE}%{WORD:was_loglevel}%{SPACE} %{GREEDYDATA:logmessage}"]
          overwrite => [ "message" ]
          #tag_on_failure => [ ]
    }
    

    if ([was_shortname] == "SystemErr") {
    	aggregate {
        task_id => "%{was_threadID}"
    		code => " map['logmessage'] ||= [] ;
                  map['logmessage'] << event.get('logmessage')"
        push_previous_map_as_event => true
    	}
    }
    else{
      grok {
          # Extract the WebSphere Response Code
          match => ["message", "(?<was_responsecode>[A-Z0-9]{9,10})[:,\s\s]"]
          tag_on_failure => [ ]
      }
      date{
                match => ["timestamp", "dd/MM/YY HH:mm:ss:SSS", "M/d/YY HH:mm:ss:SSS", "MM/d/YY HH:mm:ss:SSS", "M/dd/YY HH:mm:ss:SSS", "MM/dd/YY H:mm:ss:SSS", "M/d/YY H:mm:ss:SSS", "MM/d/YY H:mm:ss:SSS", "M/dd/YY H:mm:ss:SSS"]
      }
    }
  }

So should I place your part into

   if ([was_shortname] == "SystemErr") {
    	aggregate {
        task_id => "%{was_threadID}"

section?

Thanks.

8

No, a codec applies to an input, not a filter.

9

Our input is:

input {
  beats {
    port => 5000
  }
}

but we need filter to write our parsing sections. How can we write the same parsing within the filter aggregation?

10

If you are using filebeat to read the logs then you can do equivalent multiline processing in filebeat.

11

I appreciate if you could show me how I could write it. This syntax is not ringing any bell about how to write it in filebeat.

12

I think the filebeat equivalent of

would be

multiline.pattern: '^(\[\d{2}\.\d{2}\.\d{4} \d{2}:\d{2}:\d{2}:\d{3} \w{3}\] [0-9a-f]{8} SystemErr \w )?at'
multiline.match: after
13

Thanks Badger, that solved the main parsing problem we had.
During our testing we saw one thing that I wanted to ask you. Filebeat does parse the log according to "at" which is fine. But considering the rows below:

[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R Caused by: java.lang.NullPointerException
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at java.util.ArrayList.addAll(ArrayList.java:576)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at com.gunessigorta.pusula.uretim.UretimScriptBean.getRiskGrubuList(UretimScriptBean.java:6803)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at com.gunessigorta.pusula.uretim.UretimScriptBean.musteriYuksekRiskGrubundaMi(UretimScriptBean.java:6688)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at sun.reflect.GeneratedMethodAccessor2110.invoke(Unknown Source)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at java.lang.reflect.Method.invoke(Method.java:620)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  ... 120 more
[03.11.2020 12:02:19:850 TRT] 00000a23 SystemErr     R RollbackWrapperInterceptor.printStackTrace(): transaction rollback işaretlenecek!!! #null
[03.11.2020 12:02:19:850 TRT] 00000a23 SystemErr     R java.lang.IllegalStateException: No conversation context active
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.ScopeType.getContext(ScopeType.java:133)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.Component.getInstance(Component.java:2002)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.Component.getInstance(Component.java:1997)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.framework.service.base.ServiceUtils.getComponentInstance(ServiceUtils.java:858)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.framework.service.base.ServiceUtils.getComponentInstance(ServiceUtils.java:854)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.entegrasyon.business.impl.EntegrasyonKullanici.em(EntegrasyonKullanici.java:99)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.entegrasyon.business.impl.EntegrasyonKullanici.getKullaniciOkunmamisMesajSayisi(EntegrasyonKullanici.java:593)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.entegrasyon.business.domain.ActiveUser.getOkunmamisMesajSayisi(ActiveUser.java:326)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at java.lang.reflect.Method.invoke(Method.java:620)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.seam.RollbackWrapperInterceptor.aroundInvoke(RollbackWrapperInterceptor.java:47)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke(SynchronizationInterceptor.java:32)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:185)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:103)

This is just a portion of the log and I noticed that the parsing of those lines are done as four lines

Section 1:

[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R Caused by: java.lang.NullPointerException
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at java.util.ArrayList.addAll(ArrayList.java:576)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at com.gunessigorta.pusula.uretim.UretimScriptBean.getRiskGrubuList(UretimScriptBean.java:6803)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at com.gunessigorta.pusula.uretim.UretimScriptBean.musteriYuksekRiskGrubundaMi(UretimScriptBean.java:6688)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at sun.reflect.GeneratedMethodAccessor2110.invoke(Unknown Source)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at java.lang.reflect.Method.invoke(Method.java:620)
[03.11.2020 11:59:51:603 TRT] 000011db SystemErr     R  at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)

Section 2:

[03.11.2020 11:59:51:603 TRT] 000011db SystemErr R ... 120 more

Section 3:

[03.11.2020 12:02:19:850 TRT] 00000a23 SystemErr R RollbackWrapperInterceptor.printStackTrace(): transaction rollback işaretlenecek!!! #null

Section 4:

[03.11.2020 12:02:19:850 TRT] 00000a23 SystemErr     R java.lang.IllegalStateException: No conversation context active
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.ScopeType.getContext(ScopeType.java:133)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.Component.getInstance(Component.java:2002)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.Component.getInstance(Component.java:1997)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.framework.service.base.ServiceUtils.getComponentInstance(ServiceUtils.java:858)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.framework.service.base.ServiceUtils.getComponentInstance(ServiceUtils.java:854)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.entegrasyon.business.impl.EntegrasyonKullanici.em(EntegrasyonKullanici.java:99)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.entegrasyon.business.impl.EntegrasyonKullanici.getKullaniciOkunmamisMesajSayisi(EntegrasyonKullanici.java:593)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.entegrasyon.business.domain.ActiveUser.getOkunmamisMesajSayisi(ActiveUser.java:326)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at java.lang.reflect.Method.invoke(Method.java:620)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at com.gunessigorta.pusula.seam.RollbackWrapperInterceptor.aroundInvoke(RollbackWrapperInterceptor.java:47)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
[03.11.2020 12:02:19:852 TRT] 00000a23 SystemErr     R  at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke(SynchronizationInterceptor.java:32)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:185)
[03.11.2020 12:02:19:853 TRT] 00000a23 SystemErr     R  at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:103)

When we take a look at the thread IDs, section 3 is the beginning of section 4, so is there a way we can combine those two sections since the thread IDs

00000a23

match?

Thanks alot. I really appreciate you guidance.

14

Combining section 2 into section 1 would be easy, but using an alternation like

 (at|\.\.\.)

Combining 3 and 4 is much harder. It is not clear to me that a regexp based approach will work at all.

15

We used this for filebeat

- type: log
  paths:
    - /media/cihan/455bd60b-3793-4418-b70f-272901ff06b5/elk/logwas/SystemErr*.log
  fields: {log_type: was}
  multiline.type: pattern
  multiline.pattern: '^(\[\d{2}\.\d{2}\.\d{4} \d{2}:\d{2}:\d{2}:\d{3} \w{3}\]) (\w{8} SystemErr)     \w [[:space:]]at'
  multiline.match: after

and it worked but as I said we need to combine section 3 and 4 as well if possible.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.