Journalbeat and Java exceptions

Elastic Stack Beats
1

Hi all,

I'm trying to catch java exceptions using multiline processor as follows:

journalbeat.inputs:
- paths: []
  seek: cursor
  include_matches:
    - "syslog.identifier=tomcat"
  multiline:
    pattern: '^\d+\serror|^.+Exception: .+|^\s+at .+|^\s+... \d+ more|^\s*Caused by:.+|^[[:space:]]+|^\s*[{|}]|^(\s+)?<\/?.+>$'
    negate: false
    match: after

but the even sent has only a single line like:

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

Anyone has done this with journalbeat?

2

Ok, I don't think multiline is working properly in Journalbeat. I tested with https://play.golang.org/p/uAd5XHxscu as per https://www.elastic.co/guide/en/beats/filebeat/6.0/_test_your_regexp_pattern_for_multiline.html. This is the pattern I've tested with:

var pattern = `^\d+\serror|^.+Exception|^[[:space:]]+(at|...)|^\s+... \d+ more|^([[:space:]]+)?Caused by:.+|^\s*[{|}]|^(\s+)?<\/?.+>$`
var negate = false

and this is the exception:

var content = `2019-03-14 05:48:47,365 +0000 [-nio-8080-exec-6] ERROR MINOR  c.e.e.c.a.e.RestApiExceptionHandler - Exception 
     ... 104 common frames omitted
     at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:391)
     at org.apache.coyote.Response.doWrite(Response.java:491)
     at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
     at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
     at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(
     at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:42)
     at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:198)
     at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:140)
     at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:172)
     at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
     at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:124)
     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
     at sun.nio.ch.IOUtil.write(IOUtil.java:65)
     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
Caused by: java.io.IOException: Broken pipe
         at java.lang.Thread.run(Thread.java:748)
         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
         at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1152)
         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
         at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:542)
         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
         at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:604)
         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
         at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
         at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
         at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
         at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(
         at ty.authentication.LoggingAuthenticationErrorFilter.doFilter(LoggingAuthenticationErrorFilter.java:27)
         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(
         at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
`

and the test is successful. Journalbeat however sends only this single line:

         at java.lang.Thread.run(Thread.java:748)

Any ideas?

3

hey @igorc

unfortunately multiline support is not supported by journal beat.
If this is something you would like to see, you can still create an issue or +1 here

4

Thanks @Michal_Pristas, done.