Grok Pattern for Multiline log pattern


(Ashish) #1

from the below log line, I want to extract java.lang.exception as I need to show count of every exception in my kibana board

[28.03.19 05:44:47:954 MEZ] 0000085d SystemOut O 28 Mär 2019 05:44:47:953 [ERROR] [Server_1] Invalid request. The request must only come from a valid session.
java.lang.Exception
at ……………………………

for that I have created below logstash configuration file

input {
beats {
port => 5044
}
file {
path =>["E:\logfiles.Live*.log"]
start_position => "beginning"
codec => multiline {
pattern => "^%["
negate => true
what => "previous"
}
}

}

filter {

grok {

patterns_dir => ["E:\pattern"]

match => [ "errormessage",
           "\[(?<dateTime>%{MONTHDAY}.%{MONTHNUM}.%{YEAR} %{TIME:time} MEZ)\]%{SPACE}%{DATA:thread}\s%{WORD}\s*%{SPACE}%{USER}%{SPACE}%{MONTHDAY} %{MONTH} %{YEAR} %{TIME}%{SPACE}\[%{LOGLEVEL:loglevel}\]%{SPACE}\[%{USERNAME:Server}\]%{SPACE}\[%{SPACE}\](?:(\[?.*\]?))%{TestException:Java_Exception}(?:(\[?.*\]?))%{GREEDYDATA:messagetext}",
		   "msglog",
		   "\[(?<dateTime>%{MONTHDAY}.%{MONTHNUM}.%{YEAR} %{TIME:time} MEZ)\]%{SPACE}%{DATA:thread}\s%{WORD}\s*%{SPACE}%{USER}%{SPACE}%{MONTHDAY} %{MONTH} %{YEAR} %{TIME}%{SPACE}\[%{LOGLEVEL:loglevel}\]%{SPACE}\[%{USERNAME:Server}\]%{SPACE}\[%{SPACE}\]%{GREEDYDATA:messagetext}"
         ]

}

date {
match => [ "timestamp" , "dd.mm.yy HH:mm:ss:SSS" ]
}
}
output {
elasticsearch {
# hosts => localhost
}
stdout {}
}

In patterns folder I have created patterns.txt file with the below content
TestException java?.[.\w]+Exception

But in message field I am getting complete log message ,field Java_Exception is not created.

it is working fine on http://grokconstructor.appspot.com/do/match#result but it doesn't work when I run using config file

Any idea how to resolve this issue?