Hi.
I try to parse a log file that contains a stack trace.
I use multiline coded in order to join multiple lines in one event.
After I do this I want to use grok to identify the exception and store everything until end of line.
When I use something like this the (?<exception_details>([a-zA-Z]|.)*[a-zA-z]Exception.) , the grok will match all the event starting from the error and not until end of line.
This http://www.geocities.jp/kosako3/oniguruma/doc/RE.txt says that dot needs to match everything except end of line.
In the stdout I can see that the end of line marker is present.
Can anyone please help me understand why grok is not following the regular expression as it should?
Here is my configuration
input{
file{
path => ["d:/work/me/ELK/temp/mtc/RTSTest/RTSExceptions.*"]
start_position => beginning
add_field => {
"host" => "RTS2"
}
codec => multiline {
pattern => "^%{DATIME}"
negate => "true"
what => "previous"
}
type => "RTSExceptions"
}
}
filter {
if [type] == "RTSExceptions"{
grok{
match => ["message", "%{DATIME:mytimestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}:%{SPACE}%{BLOCK:thread}(?.)"]
}
grok{
match => ["message", "(?([a-zA-Z]|.)[a-zA-z]Exception)"]
}
grok{
match => ["message", "(?<exception_details>([a-zA-Z]|.)[a-zA-z]Exception.)"]
}
date{
match => ["mytimestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
target => "@timestamp"
timezone => "UTC"
remove_field => ["mytimestamp"]
}
}
}
Here is the stdout when I run logstash:
{
"@timestamp" => "2015-04-10T14:39:52.787Z",
"message" => "2015-04-10 14:39:52,787 FATAL: [RTSRequestHandler-11] [RTSRequestHandler] RTSRequestHandler.handleRegularMode. LogID=RTS_172.20.17.102_109940522_14232\r\njava.sql.SQLException: com.mind.utils.exceptions.MindTypeException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:1257)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:618)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.InRequest.callEnd(InRequest.java:502)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.voice.InApplyChargingReportRequest.a(InApplyChargingReportRequest.java:56)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.voice.InApplyChargingReportRequest.process(InApplyChargingReportRequest.java:168)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.RTSTransaction.process(RTSTransaction.java:117)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.a(RTSRequestHandler.java:106)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.d(RTSRequestHandler.java:69)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.work(RTSRequestHandler.java:199)\r\n\tat com.mind.utils.threads.messagethread.reconnectthread.ReconnectThread.execute(ReconnectThread.java:220)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.execute(RTSRequestHandler.java:75)\r\n\tat com.mind.utils.threads.BaseThread.run(BaseThread.java:239)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: com.mind.utils.exceptions.MindTypeException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat com.mind.utils.connection.db.BaseStatement.error(BaseStatement.java:175)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:199)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:165)\r\n\tat com.mind.finance.engine.rc.logic.actions.GetAccountHierarchyLock.a(GetAccountHierarchyLock.java:13)\r\n\tat com.mind.finance.engine.rc.logic.actions.GetAccountHierarchyLock.perform(GetAccountHierarchyLock.java:21)\r\n\tat com.mind.finance.engine.rc.logic.RCCalc.f(RCCalc.java:730)\r\n\tat com.mind.finance.engine.rc.logic.RCCalc.calculate(RCCalc.java:972)\r\n\tat com.mind.finance.engine.rc.RCEngine.calculate(RCEngine.java:140)\r\n\tat com.mind.mediation.common.logic.processing.ServiceBaseProcess.calcRCCharges(ServiceBaseProcess.java:3311)\r\n\tat com.mind.mediation.common.logic.processing.voip.ServiceVoipProcess.addCDR(ServiceVoipProcess.java:8092)\r\n\tat com.mind.mediation.common.logic.processing.voip.ServiceVoipProcess.addCallToDB(ServiceVoipProcess.java:8681)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:248)\r\n\t... 12 more\r\nCaused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)\r\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)\r\n\tat oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1017)\r\n\tat oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:655)\r\n\tat oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:249)\r\n\tat oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:566)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:210)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:53)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:938)\r\n\tat oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1075)\r\n\tat oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3820)\r\n\tat oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3923)\r\n\tat oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:5617)\r\n\tat oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1385)\r\n\tat com.mind.utils.connection.db.Procedure.doExecute(Procedure.java:1801)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:179)\r\n\t... 22 more\r",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"type" => "RTSExceptions",
"host" => "RTS2",
"path" => "d:/work/me/ELK/temp/mtc/RTSTest/RTSExceptions.log",
"loglevel" => "FATAL",
"thread" => "[RTSRequestHandler-11]",
"content" => " [RTSRequestHandler] RTSRequestHandler.handleRegularMode. LogID=RTS_172.20.17.102_109940522_14232\r\njava.sql.SQLException: com.mind.utils.exceptions.MindTypeException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:1257)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:618)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.InRequest.callEnd(InRequest.java:502)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.voice.InApplyChargingReportRequest.a(InApplyChargingReportRequest.java:56)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.voice.InApplyChargingReportRequest.process(InApplyChargingReportRequest.java:168)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.RTSTransaction.process(RTSTransaction.java:117)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.a(RTSRequestHandler.java:106)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.d(RTSRequestHandler.java:69)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.work(RTSRequestHandler.java:199)\r\n\tat com.mind.utils.threads.messagethread.reconnectthread.ReconnectThread.execute(ReconnectThread.java:220)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.execute(RTSRequestHandler.java:75)\r\n\tat com.mind.utils.threads.BaseThread.run(BaseThread.java:239)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: com.mind.utils.exceptions.MindTypeException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat com.mind.utils.connection.db.BaseStatement.error(BaseStatement.java:175)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:199)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:165)\r\n\tat com.mind.finance.engine.rc.logic.actions.GetAccountHierarchyLock.a(GetAccountHierarchyLock.java:13)\r\n\tat com.mind.finance.engine.rc.logic.actions.GetAccountHierarchyLock.perform(GetAccountHierarchyLock.java:21)\r\n\tat com.mind.finance.engine.rc.logic.RCCalc.f(RCCalc.java:730)\r\n\tat com.mind.finance.engine.rc.logic.RCCalc.calculate(RCCalc.java:972)\r\n\tat com.mind.finance.engine.rc.RCEngine.calculate(RCEngine.java:140)\r\n\tat com.mind.mediation.common.logic.processing.ServiceBaseProcess.calcRCCharges(ServiceBaseProcess.java:3311)\r\n\tat com.mind.mediation.common.logic.processing.voip.ServiceVoipProcess.addCDR(ServiceVoipProcess.java:8092)\r\n\tat com.mind.mediation.common.logic.processing.voip.ServiceVoipProcess.addCallToDB(ServiceVoipProcess.java:8681)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:248)\r\n\t... 12 more\r\nCaused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)\r\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)\r\n\tat oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1017)\r\n\tat oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:655)\r\n\tat oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:249)\r\n\tat oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:566)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:210)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:53)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:938)\r\n\tat oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1075)\r\n\tat oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3820)\r\n\tat oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3923)\r\n\tat oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:5617)\r\n\tat oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1385)\r\n\tat com.mind.utils.connection.db.Procedure.doExecute(Procedure.java:1801)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:179)\r\n\t... 22 more\r",
"exception" => "java.sql.SQLException",
"exception_details" => "java.sql.SQLException: com.mind.utils.exceptions.MindTypeException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:1257)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:618)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.InRequest.callEnd(InRequest.java:502)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.voice.InApplyChargingReportRequest.a(InApplyChargingReportRequest.java:56)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.in.voice.InApplyChargingReportRequest.process(InApplyChargingReportRequest.java:168)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.protocols.RTSTransaction.process(RTSTransaction.java:117)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.a(RTSRequestHandler.java:106)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.d(RTSRequestHandler.java:69)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.work(RTSRequestHandler.java:199)\r\n\tat com.mind.utils.threads.messagethread.reconnectthread.ReconnectThread.execute(ReconnectThread.java:220)\r\n\tat com.mind.mediation.applications.realtimeserver.modules.processing.RTSRequestHandler.execute(RTSRequestHandler.java:75)\r\n\tat com.mind.utils.threads.BaseThread.run(BaseThread.java:239)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: com.mind.utils.exceptions.MindTypeException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat com.mind.utils.connection.db.BaseStatement.error(BaseStatement.java:175)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:199)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:165)\r\n\tat com.mind.finance.engine.rc.logic.actions.GetAccountHierarchyLock.a(GetAccountHierarchyLock.java:13)\r\n\tat com.mind.finance.engine.rc.logic.actions.GetAccountHierarchyLock.perform(GetAccountHierarchyLock.java:21)\r\n\tat com.mind.finance.engine.rc.logic.RCCalc.f(RCCalc.java:730)\r\n\tat com.mind.finance.engine.rc.logic.RCCalc.calculate(RCCalc.java:972)\r\n\tat com.mind.finance.engine.rc.RCEngine.calculate(RCEngine.java:140)\r\n\tat com.mind.mediation.common.logic.processing.ServiceBaseProcess.calcRCCharges(ServiceBaseProcess.java:3311)\r\n\tat com.mind.mediation.common.logic.processing.voip.ServiceVoipProcess.addCDR(ServiceVoipProcess.java:8092)\r\n\tat com.mind.mediation.common.logic.processing.voip.ServiceVoipProcess.addCallToDB(ServiceVoipProcess.java:8681)\r\n\tat com.mind.mediation.applications.realtimeserver.logic.processing.RTSVoipProcess.callEnd(RTSVoipProcess.java:248)\r\n\t... 12 more\r\nCaused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation\r\nORA-06512: at "SYS.DBMS_LOCK", line 82\r\nORA-06512: at "SYS.DBMS_LOCK", line 98\r\nORA-06512: at "MINDV7.MIND_GENERAL", line 20\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4572\r\nORA-06512: at "MINDV7.MIND_CORRELATION", line 4676\r\nORA-06512: at line 1\r\n\r\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)\r\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)\r\n\tat oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1017)\r\n\tat oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:655)\r\n\tat oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:249)\r\n\tat oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:566)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:210)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:53)\r\n\tat oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:938)\r\n\tat oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1075)\r\n\tat oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3820)\r\n\tat oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3923)\r\n\tat oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:5617)\r\n\tat oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1385)\r\n\tat com.mind.utils.connection.db.Procedure.doExecute(Procedure.java:1801)\r\n\tat com.mind.utils.connection.db.Procedure.execute(Procedure.java:179)\r\n\t... 22 more\r"
}