Multiline Java Stack Traces with Syslog

[taytos (Adam)]

Hello there!

I have researched this problem I'm having for a long time and can't seem to figure it out. I'm trying to avoid using the multiline filter plug-in as that is being deprecated so the multiline input codec is being used instead. Here is my input config:

input {
    tcp {
      port => 5514
      type => syslog
      codec => multiline {
        pattern => "%{SYSLOG5424PRI}%{SYSLOGBASE2}%{SPACE}%{JAVASTACKTRACEPART}"
        what => "previous"
      }
    }
}

Note that I am not using the syslog input plugin. I have tried and it only seems to make it worse for one reason or another. Also the logs are being sent via nxlog. That is something that I cannot change unfortunately. Any who, the results I get from this are not quiet what I'm looking for:

at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
<11>Feb 16 10:41:00 myserver selfserve: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
<11>Feb 16 10:41:00 myserver selfserve: 	at java.lang.reflect.Method.invoke(Method.java:498)
<11>Feb 16 10:41:00 myserver selfserve: 	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
<11>Feb 16 10:41:00 myserver selfserve: 	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
<11>Feb 16 10:41:00 myserver selfserve: 	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
<11>Feb 16 10:41:00 myserver selfserve: 	at com.codahale.metrics.jersey.InstrumentedResourceMethodDispatchProvider$TimedRequestDispatcher.dispatch(InstrumentedResourceMethodDispatchProvider.java:30)
<11>Feb 16 10:41:00 myserver selfserve: 	at io.dropwizard.jersey.guava.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(OptionalResourceMethodDispatchAdapter.java:37)
<11>Feb 16 10:41:00 myserver selfserve: 	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
<11>Feb 16 10:41:00 myserver selfserve: 	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
<11>Feb 16 10:41:00 myserver selfserve: 	at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)

It starts out OK (first line) but every line after gets the additional info that is appended because shipping as syslog. It also does not seem to include the line that states the exception before the stack trace. I even try to grok this to see if it will help but I get the same results. Here is my filter config:

filter {
  if [type] == "syslog" {
    grok {
       match => [ "message", "%{SYSLOG5424PRI}%{SYSLOGBASE2} %{GREEDYDATA:log_message}" ]
    }
  }
}

And then just sending to a local Elasticsearch instance and viewing the logs with Kibana.

Anyone have any thoughts or want additional details?

Thanks!

[taytos (Adam)]

Anyone? I tried sending the logs without syslog and the multiline works OK but I lose out on some good info that syslog appends like the program name that is sending the logs. Bummer! I'm curious if I were to send these logs to a logstash instance that groks the logs so I get a message field with just the log data and stashes the data in Redis, then send it to another logstash instance that takes Redis input and does the mulitline input codec? Not sure if that will do the trick but might be worth a shot. Any input would be greatly appreciated!