Parse logs from log4j to logstash

Elastic Stack Logstash
1

I'm trying to parse some old log files from log4j, but I can't find the right pattern for grok to do that.

This is a snipet from the log

2018-04-12T15:10:25.180-0300 INFO  CORE:VinculoServico:49 - teste
2018-04-12T15:10:29.065-0300 INFO  SegurancaFiltro:47 - ### SegurancaFiltro: URL
2018-04-12T15:10:29.105-0300 INFO  AutorizadorServicos:233 - ### Usuário: 123
2018-04-12T15:10:29.120-0300 INFO  LocalizadorServico:58 - buscando servico [ /arquivos-download-api/api ]
2018-04-12T15:10:29.123-0300 INFO  LocalizadorServico:30 - servico localizado [ /arquivos-download-api/api ]
2018-04-12T15:10:29.191-0300 INFO  AutorizadorServicos:83 - URL Autorizada [ Usuario [id=123, email=teste@teste.com], urlRedirect:/arquivos-download-api/api, mensagem: Acesso Autorizado com sucesso. ]
java.nio.file.FileSystemException: teste: Not a directory
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
	at java.nio.file.Files.newByteChannel(Files.java:361)
	at java.nio.file.Files.newByteChannel(Files.java:407)
	at java.nio.file.Files.readAllBytes(Files.java:3152)
	at br.gov.pa.sefa.arquivosdownload2.core.downloads.DownloadsServico.downloadArquivo(DownloadsServico.java:29)
	at br.gov.pa.sefa.arquivosdownload2.core.downloads.DownloadsServico$$FastClassBySpringCGLIB$$548010da.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:652)
	at br.gov.pa.sefa.arquivosdownload2.core.downloads.DownloadsServico$$EnhancerBySpringCGLIB$$d9c365ec.downloadArquivo(<generated>)
	at br.gov.pa.sefa.arquivosdownload2.api.rest.downloads.DownloadsRecurso.downloadArquivo(DownloadsRecurso.java:42)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
	at br.gov.pa.sefa.autorizador.filtros.SegurancaFiltro.doFilter(SegurancaFiltro.java:54)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
	at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3431)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3397)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
	at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
	at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1575)
	at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
2018-04-12T15:10:33.324-0300 INFO  SegurancaFiltro:47 - ### SegurancaFiltro: URL

And this is my filter configuration

filter {
  ## WebLogic Server Log
  if "wlsserverlog" in [tags] {
    grok {
      match => [ "message", "(?<timestamp>%{YEAR}-%{MONTHNUM2}-%{MONTHDAY}THH:mm:ss.SSSZ) %{LOGLEVEL:loglevel} * (?<logger>[A-Za-z0-9$_.]+):%{NONNEGINT:line} - %{GREEDYDATA:message}$" ]
    }
  }
}
2

What would you like the end result to be? You have log entries that seem to have a single line with a timestamp, level, and message, and other log entries that seem to have a timestamp, level, message, followed by a multiline stacktrace. Do you want each of these to be a single event?

The documentation has examples of using multiline within beats or logstash to combine lines in a java stack trace.

What does the input section of the logstash configuration look like?

3

Yeah, I would like each of these to be a single event. I'm not using beats, only logstash. This is my input section:

input {
  ## WebLogic Server Log
  file {
    type => "weblogic"
    path => "path"
	start_position => "beginning"
    sincedb_path => "/dev/null"
	ignore_older => 0 
    tags => ["wlsserverlog"]
  }
4

OK, so you need a multiline codec on the file input. If you are OK with the whole exception being an event on its own then this would work (that's a tab in there after the ^).

codec => multiline { pattern => "^        " negate => false what => "previous" auto_flush_interval => 3

However, that does not get you a timestamp on the exception, so you might be better off with

codec => multiline { pattern => "^2" negate => true what => "previous" auto_flush_interval => 3

Which is fragile, since it depends on the date starting with a 2, but it works for the next thousand years, which is good enough for me.

I would use dissect to pull everything up to the first space into a timestamp field, then your pattern works just fine for everything except the timestamp.

  dissect { mapping => { "message" => "%{timestamp} %{[@metadata][notTheTimestamp]}" } }
  date { match => [ "timestamp", "ISO8601" ] }
  grok {
    match => [ "[@metadata][notTheTimestamp]", "%{LOGLEVEL:loglevel} * (?<logger>[A-Za-z0-9$_.]+):%{NONNEGINT:line} - %{GREEDYDATA:logmessage}$" ]
  }
#  mutate { remove_field => [ "message" ] }
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.