# Filebeat Multiline Java Stack Trace
**URL:** https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003
**Category:** Beats
**Tags:** filebeat
**Created:** [September 26, 2018, 12:19pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003 "2018-09-26T12:19:02Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@luggo](https://discuss.elastic.co/u/luggo)
#### Post date: [September 26, 2018, 12:19pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/1 "2018-09-26T12:19:02Z")
Hey there,
I try to find out how to use Filebeat for my Java Log files. Actually it's not a big deal, except for my problems with multiline messages, because my Java Logs include Stack Traces.
Can someone help me to find the right pattern for my case?
Currently I am using this pattern, which combines all rows starting with spaces and "Caused by" (from [this](https://www.elastic.co/guide/en/beats/filebeat/current/_examples_of_multiline_configuration.html) example). But unfortunately it does not combine the Stack Trace with the Java Log, instead it just combines the Stack Trace to one message, leaving the Java Error Log without Stack Trace.
**Filebeat config**
```
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
```
**Exampe Java Log with Stacktrace**
```
2018-05-31 08:44:22,771 ERROR com.zettler.ec.jsf.common.security.spring.ZettlerAuthenticationProcessingFilter - An internal error occurred while trying to authenticate the user.
org.springframework.security.authentication.InternalAuthenticationServiceException: User ist in Datenbank nicht angelegt.
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:110)
at org.springframework.security.authentication.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:132)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156)
at com.zettler.ec.jsf.common.security.spring.ZettlerAuthenticationProcessingFilter.attemptAuthentication(ZettlerAuthenticationProcessingFilter.java:89)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:211)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: User ist in Datenbank nicht angelegt.
at com.zettler.ec.jsf.common.security.spring.MfxUserDetailsService.loadUserByUsername(MfxUserDetailsService.java:80)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:102)
... 34 more
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 26, 2018, 12:33pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/2 "2018-09-26T12:33:05Z")
Does the [timestamp example in the documentation](https://www.elastic.co/guide/en/beats/filebeat/6.4/_examples_of_multiline_configuration.html#_timestamps) now work for this?
---
### Author:  [@luggo](https://discuss.elastic.co/u/luggo)
#### Post date: [September 26, 2018, 12:39pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/3 "2018-09-26T12:39:42Z")
It does! But it does not add the Stack Trace to the previous Java Error Log.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 26, 2018, 12:46pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/4 "2018-09-26T12:46:21Z")
I am not sure I follow. Can you show a full example?
---
### Author:  [@luggo](https://discuss.elastic.co/u/luggo)
#### Post date: [September 26, 2018, 1:06pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/5 "2018-09-26T13:06:13Z")
No problem, I am sorry for my unprecise description... this is my first IT project.
I use an Filebeats \> Logstash \> Elasticsearch (\> Kibana) configuration.
The idea is, to let Filebeats extract the logs from the hundreds of .log-files we have in our storages and send it to Logstash, where it gets mapped/ structured in the for Elasticsearch appropriate way.
Now (for testing purposes) my Logstash receives the messages of the Filebeat and prints them in the console (stdout).
When I'm starting filebeat (the multiline.pattern) and logstash, then logstash prints out for each Error Stack Trace Log, two different JSON messages. One containing the first part (Error Log):
```
2018-05-31 21:27:33,065 ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/mfx##20180524121538_19.1.15.4243].[AutoRegisteredResourceServlet] - Servlet.service() for servlet [AutoRegisteredResourceServlet] in context with path [/mfx##20180524121538_19.1.15.4243] threw exception
```
The other one, containing the second part (Stack Trace):
```
java.lang.IllegalStateException: getAttribute: Session already invalidated
at org.apache.catalina.session.StandardSession.getAttribute(StandardSession.java:1191)
at org.apache.catalina.session.StandardSessionFacade.getAttribute(StandardSessionFacade.java:103)
at base.web.session_countdown.HttpSessionRefreshFilter.getSessionAttribute(HttpSessionRefreshFilter.java:134)
at base.web.session_countdown.HttpSessionRefreshFilter.doRefreshActions(HttpSessionRefreshFilter.java:82)
at base.web.session_countdown.HttpSessionRefreshFilter.doFilter(HttpSessionRefreshFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
```
For me, it would be better to have the whole Stack Trace togheter with the Error Log.
Thank you so much for your time!
---
### Author:  [@KumarSaurabh](https://discuss.elastic.co/u/KumarSaurabh)
#### Post date: [September 26, 2018, 2:25pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/6 "2018-09-26T14:25:56Z")
I am also following response closely as I tried to do this but could not. Thanks for posting it
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 26, 2018, 2:37pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/7 "2018-09-26T14:37:41Z")
The pattern I linked to, which looks for a line starting with a timestamp and adds all subsequent lines that do not should as far as I understand handle that and group it the way you want.
---
### Author:  [@luggo](https://discuss.elastic.co/u/luggo)
#### Post date: [September 27, 2018, 8:38am UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/8 "2018-09-27T08:38:55Z")
Thank you Christian.
Yes, this was actually also my first tought. I tried it, but all what happened is that the whole Log file was grouped to one message.
Now I tried:
```
multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
```
And it worked out, except for the last line (log), which was "ignored", because it was not printed in the console.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 27, 2018, 8:41am UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/9 "2018-09-27T08:41:13Z")
This should be sent on once it has timed out, which should happen after 5 seconds by default.
---
### Author:  [@luggo](https://discuss.elastic.co/u/luggo)
#### Post date: [September 27, 2018, 8:48am UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/10 "2018-09-27T08:48:34Z")
Actually nothing happens, even when I define that `multiline.timeout: 5s`.
---
### Author:  [@luggo](https://discuss.elastic.co/u/luggo)
#### Post date: [October 10, 2018, 12:13pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/12 "2018-10-10T12:13:39Z")
So, after 2 weeks I still haven't found the right solution to my problem.
**Example Error Log**
```
2018-05-31 08:44:22,771 ERROR com.zettler.ec.jsf.common.security.spring.ZettlerAuthenticationProcessingFilter - An internal error occurred while trying to authenticate the user.
org.springframework.security.authentication.InternalAuthenticationServiceException: User ist in Datenbank nicht angelegt.
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: User ist in Datenbank nicht angelegt.
at com.zettler.ec.jsf.common.security.spring.MfxUserDetailsService.loadUserByUsername(MfxUserDetailsService.java:80)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:102)
... 34 more
```
**Current state**
What the following configuration does, is that it merges any line that begins with whitespace up to the previous line.
```auto
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
multiline.negate: false
multiline.match: after
```
`The problem is, that I would have now two separated "Logs" or "Events" (the product of the Filebeat) for actually one Error Log.`
In the case of the example error log from above, the Filebeat would do the following...
One log with (Java Log):
```
2018-05-31 08:44:22,771 ERROR com.zettler.ec.jsf.common.security.spring.ZettlerAuthenticationProcessingFilter - An internal error occurred while trying to authenticate the user.
```
And, one log with (the Stack Trace belonging to the Java Log):
```
org.springframework.security.authentication.InternalAuthenticationServiceException: User ist in Datenbank nicht angelegt.
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: User ist in Datenbank nicht angelegt.
at com.zettler.ec.jsf.common.security.spring.MfxUserDetailsService.loadUserByUsername(MfxUserDetailsService.java:80)
at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:102)
... 34 more
```
**Target state**
Having a configuration that merges all the Stack Trace lines to the belonging Error Log - having 1 Log for each Log and not 2, in case of Logs with Stack Traces.
.
I hope that my problem is more clear now.
---
### Author:  [@marix](https://discuss.elastic.co/u/marix)
#### Post date: [November 4, 2018, 3:55pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/13 "2018-11-04T15:55:16Z")
try it，be care for " **^**"
**multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'**
**multiline.negate: true**
**multiline.match: after**
if delay to long or log to long，add these options
**multiline.timeout: 120s** #default 5s
**multiline.max\_lines: 10000** #default 500
more about options see this [https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)
---
### Author:  [@luggo](https://discuss.elastic.co/u/luggo)
#### Post date: [November 7, 2018, 9:37am UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/14 "2018-11-07T09:37:36Z")
Thank you, that worked out!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 5, 2018, 9:37am UTC](https://discuss.elastic.co/t/filebeat-multiline-java-stack-trace/150003/15 "2018-12-05T09:37:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.