Parse logs from log4j to logstash

(Bruno Andrade) #1

I'm trying to parse some old log files from log4j, but I can't find the right pattern for grok to do that.

This is a snipet from the log

2018-04-12T15:10:25.180-0300 INFO  CORE:VinculoServico:49 - teste
2018-04-12T15:10:29.065-0300 INFO  SegurancaFiltro:47 - ### SegurancaFiltro: URL
2018-04-12T15:10:29.105-0300 INFO  AutorizadorServicos:233 - ### Usuário: 123
2018-04-12T15:10:29.120-0300 INFO  LocalizadorServico:58 - buscando servico [ /arquivos-download-api/api ]
2018-04-12T15:10:29.123-0300 INFO  LocalizadorServico:30 - servico localizado [ /arquivos-download-api/api ]
2018-04-12T15:10:29.191-0300 INFO  AutorizadorServicos:83 - URL Autorizada [ Usuario [id=123,], urlRedirect:/arquivos-download-api/api, mensagem: Acesso Autorizado com sucesso. ]
java.nio.file.FileSystemException: teste: Not a directory
	at sun.nio.fs.UnixException.translateToIOException(
	at sun.nio.fs.UnixException.rethrowAsIOException(
	at sun.nio.fs.UnixException.rethrowAsIOException(
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(
	at java.nio.file.Files.newByteChannel(
	at java.nio.file.Files.newByteChannel(
	at java.nio.file.Files.readAllBytes(
	at org.springframework.cglib.proxy.MethodProxy.invoke(
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(
	at java.lang.reflect.Method.invoke(
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(
	at org.springframework.web.servlet.DispatcherServlet.doService(
	at org.springframework.web.servlet.FrameworkServlet.processRequest(
	at org.springframework.web.servlet.FrameworkServlet.doGet(
	at javax.servlet.http.HttpServlet.service(
	at org.springframework.web.servlet.FrameworkServlet.service(
	at javax.servlet.http.HttpServlet.service(
	at weblogic.servlet.internal.StubSecurityHelper$
	at weblogic.servlet.internal.StubSecurityHelper$
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(
	at weblogic.servlet.internal.ServletStubImpl.execute(
	at weblogic.servlet.internal.TailFilter.doFilter(
	at weblogic.servlet.internal.FilterChainImpl.doFilter(
	at weblogic.servlet.internal.FilterChainImpl.doFilter(
	at weblogic.servlet.internal.RequestEventsFilter.doFilter(
	at weblogic.servlet.internal.FilterChainImpl.doFilter(
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(
	at weblogic.servlet.internal.WebAppServletContext$
	at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(
	at weblogic.servlet.internal.WebAppServletContext.execute(
	at weblogic.servlet.provider.ContainerSupportProviderImpl$
2018-04-12T15:10:33.324-0300 INFO  SegurancaFiltro:47 - ### SegurancaFiltro: URL

And this is my filter configuration

filter {
  ## WebLogic Server Log
  if "wlsserverlog" in [tags] {
    grok {
      match => [ "message", "(?<timestamp>%{YEAR}-%{MONTHNUM2}-%{MONTHDAY}THH:mm:ss.SSSZ) %{LOGLEVEL:loglevel} * (?<logger>[A-Za-z0-9$_.]+):%{NONNEGINT:line} - %{GREEDYDATA:message}$" ]


What would you like the end result to be? You have log entries that seem to have a single line with a timestamp, level, and message, and other log entries that seem to have a timestamp, level, message, followed by a multiline stacktrace. Do you want each of these to be a single event?

The documentation has examples of using multiline within beats or logstash to combine lines in a java stack trace.

What does the input section of the logstash configuration look like?

(Bruno Andrade) #3

Yeah, I would like each of these to be a single event. I'm not using beats, only logstash. This is my input section:

input {
  ## WebLogic Server Log
  file {
    type => "weblogic"
    path => "path"
	start_position => "beginning"
    sincedb_path => "/dev/null"
	ignore_older => 0 
    tags => ["wlsserverlog"]


OK, so you need a multiline codec on the file input. If you are OK with the whole exception being an event on its own then this would work (that's a tab in there after the ^).

codec => multiline { pattern => "^        " negate => false what => "previous" auto_flush_interval => 3

However, that does not get you a timestamp on the exception, so you might be better off with

codec => multiline { pattern => "^2" negate => true what => "previous" auto_flush_interval => 3

Which is fragile, since it depends on the date starting with a 2, but it works for the next thousand years, which is good enough for me.

I would use dissect to pull everything up to the first space into a timestamp field, then your pattern works just fine for everything except the timestamp.

  dissect { mapping => { "message" => "%{timestamp} %{[@metadata][notTheTimestamp]}" } }
  date { match => [ "timestamp", "ISO8601" ] }
  grok {
    match => [ "[@metadata][notTheTimestamp]", "%{LOGLEVEL:loglevel} * (?<logger>[A-Za-z0-9$_.]+):%{NONNEGINT:line} - %{GREEDYDATA:logmessage}$" ]
#  mutate { remove_field => [ "message" ] }

(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.