# Parse logs from log4j to logstash **URL:** \ **Category:** Logstash\ **Created:** [May 9, 2018, 12:58am UTC](https://discuss.elastic.co/t/parse-logs-from-log4j-to-logstash/131082 "2018-05-09T00:58:36Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![andradedb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andradedb/32/30952_2.png) [@andradedb](https://discuss.elastic.co/u/andradedb)\ **Post date:** [May 9, 2018, 12:58am UTC](https://discuss.elastic.co/t/parse-logs-from-log4j-to-logstash/131082/1 "2018-05-09T00:58:36Z")
I'm trying to parse some old log files from log4j, but I can't find the right pattern for grok to do that. This is a snipet from the log ``` 2018-04-12T15:10:25.180-0300 INFO CORE:VinculoServico:49 - teste 2018-04-12T15:10:29.065-0300 INFO SegurancaFiltro:47 - ### SegurancaFiltro: URL 2018-04-12T15:10:29.105-0300 INFO AutorizadorServicos:233 - ### Usuário: 123 2018-04-12T15:10:29.120-0300 INFO LocalizadorServico:58 - buscando servico [/arquivos-download-api/api] 2018-04-12T15:10:29.123-0300 INFO LocalizadorServico:30 - servico localizado [/arquivos-download-api/api] 2018-04-12T15:10:29.191-0300 INFO AutorizadorServicos:83 - URL Autorizada [Usuario [id=123, email=teste@teste.com], urlRedirect:/arquivos-download-api/api, mensagem: Acesso Autorizado com sucesso. ] java.nio.file.FileSystemException: teste: Not a directory at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) at java.nio.file.Files.newByteChannel(Files.java:361) at java.nio.file.Files.newByteChannel(Files.java:407) at java.nio.file.Files.readAllBytes(Files.java:3152) at br.gov.pa.sefa.arquivosdownload2.core.downloads.DownloadsServico.downloadArquivo(DownloadsServico.java:29) at br.gov.pa.sefa.arquivosdownload2.core.downloads.DownloadsServico$$FastClassBySpringCGLIB$$548010da.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:652) at br.gov.pa.sefa.arquivosdownload2.core.downloads.DownloadsServico$$EnhancerBySpringCGLIB$$d9c365ec.downloadArquivo() at br.gov.pa.sefa.arquivosdownload2.api.rest.downloads.DownloadsRecurso.downloadArquivo(DownloadsRecurso.java:42) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:844) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) at br.gov.pa.sefa.autorizador.filtros.SegurancaFiltro.doFilter(SegurancaFiltro.java:54) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3431) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3397) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57) at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1575) at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311) at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) 2018-04-12T15:10:33.324-0300 INFO SegurancaFiltro:47 - ### SegurancaFiltro: URL ``` And this is my filter configuration ``` filter { ## WebLogic Server Log if "wlsserverlog" in [tags] { grok { match => ["message", "(?%{YEAR}-%{MONTHNUM2}-%{MONTHDAY}THH:mm:ss.SSSZ) %{LOGLEVEL:loglevel} * (?[A-Za-z0-9$_.]+):%{NONNEGINT:line} - %{GREEDYDATA:message}$" ] } } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 9, 2018, 1:23am UTC](https://discuss.elastic.co/t/parse-logs-from-log4j-to-logstash/131082/2 "2018-05-09T01:23:47Z")
What would you like the end result to be? You have log entries that seem to have a single line with a timestamp, level, and message, and other log entries that seem to have a timestamp, level, message, followed by a multiline stacktrace. Do you want each of these to be a single event? The documentation has examples of using multiline within beats or logstash to combine lines in a java stack trace. What does the input section of the logstash configuration look like? ---
**Author:** ![andradedb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andradedb/32/30952_2.png) [@andradedb](https://discuss.elastic.co/u/andradedb)\ **Post date:** [May 9, 2018, 1:35am UTC](https://discuss.elastic.co/t/parse-logs-from-log4j-to-logstash/131082/3 "2018-05-09T01:35:48Z")
Yeah, I would like each of these to be a single event. I'm not using beats, only logstash. This is my input section: ``` input { ## WebLogic Server Log file { type => "weblogic" path => "path" start_position => "beginning" sincedb_path => "/dev/null" ignore_older => 0 tags => ["wlsserverlog"] } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 9, 2018, 1:19pm UTC](https://discuss.elastic.co/t/parse-logs-from-log4j-to-logstash/131082/4 "2018-05-09T13:19:46Z")
OK, so you need a multiline codec on the file input. If you are OK with the whole exception being an event on its own then this would work (that's a tab in there after the ^). ```auto codec => multiline { pattern => "^ " negate => false what => "previous" auto_flush_interval => 3 ``` However, that does not get you a timestamp on the exception, so you might be better off with ```auto codec => multiline { pattern => "^2" negate => true what => "previous" auto_flush_interval => 3 ``` Which is fragile, since it depends on the date starting with a 2, but it works for the next thousand years, which is good enough for me. I would use dissect to pull everything up to the first space into a timestamp field, then your pattern works just fine for everything except the timestamp. ```auto dissect { mapping => { "message" => "%{timestamp} %{[@metadata][notTheTimestamp]}" } } date { match => ["timestamp", "ISO8601"] } grok { match => ["[@metadata][notTheTimestamp]", "%{LOGLEVEL:loglevel} * (?[A-Za-z0-9$_.]+):%{NONNEGINT:line} - %{GREEDYDATA:logmessage}$" ] } # mutate { remove_field => ["message"] } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 6, 2018, 1:19pm UTC](https://discuss.elastic.co/t/parse-logs-from-log4j-to-logstash/131082/5 "2018-06-06T13:19:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.