Matching a field (exception name) on the second line in a multiline event

Elastic Stack Logstash
1

I have multiline Log4J logs. The lines after the first are optional. The first line contains the log message and after that is the stacktrace. I want to extract the top exception class name on a stacktrace and it is the first part on the second line. An example:

2016-01-18 13:19:34,812 [myScheduler-4] ERROR com.company.framework.service.notification.TriggerServiceImpl- Hibernate operation: could not load an entity: [com.company.framework.pojo.jc3iedm.ReportingData#32300000000000565988];.
java.sql.SQLException: Connection has already been closed.
	at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:118)
	at com.sun.proxy.$Proxy47.prepareStatement(Unknown Source)
	at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)

I'm interested on the 'java.sql.SQLException' part. I wrote the following grok pattern:

LOG4J_DATESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})
LOG4J_LOG %{LOG4J_DATESTAMP:timestamp} \[%{GREEDYDATA:thread}\] {LOGLEVEL:level} %{JAVACLASS:class}\-%{SPACE}%{JAVALOGMESSAGE:logmessage}%{SPACE}(^%{JAVACLASS:exception})?

(^%{JAVACLASS:exception})? is the part, which should match the next line. It matches as expected on the grokconstructor.appspot.com but i can't get it working with the logstash 2.1.1 version. All other fields are extracted successfully.

This is the logstash conf file:

input {
 file {
	path => "D:/projects/ELK/localhost.log"
	start_position => beginning
	codec => multiline {
	  patterns_dir => "../patterns"
      pattern => "^%{LOG4J_DATESTAMP}"
      negate => true
      what => "previous"
    }
 }
}

filter {
 grok {
    match => { "message" => "%{LOG4J_LOG}"}
 }
}

output { 
  elasticsearch { }
}

My guess is that logmessage ( JAVALOGMESSAGE (.*) ) matches everything up to the end and therefore there is nothing left to match. But my questions are how is it working on the grokconstructor.appspot.com? and how to achieve this with logstash?

thanks.
ilhami

2

I suspect you have to start the regexp with (?m) so that newline characters don't count as the end of the string.

3

i think (?m) does just the opposite. Via this mode, dot (.) starts to match the newline too. See this please https://logstash.jira.com/browse/LOGSTASH-1097

4

Ah, I didn't see that you were trying to use ^ to match the beginning of the next line. In that case I don't know.