Grok Pattern works in Grok debugger but fails in the parsing

Elastic Stack Logstash
1

The Grok Patter works in Grok debugger but fails to parse in logstash. I need help not sure whats wrong, I need to sort out ERROR logs separate Index and regular Debug into different index, My trying to get parsing done, but not sure, need some help.

When I test the patter in debugger it works
{
"package": "util.IConveUtility ",
"javaclass": "DAWSConnThread",
"log": " domainInitRqst::1::null::DDS_DATA_SOURCE_UNAVAILABLE::[[ERROR,Participant Error:null]]",
"action": " DAError",
"LEVEL": "ERROR",
"timestamp": "2019-07-03 03:06:10,043"
}

Input data

DEBUG 2019-07-03 02:58:42,024 [main] util.IConveUtility - Files are deleted..
DEBUG 2019-07-03 02:58:42,024 [main] util.IConveUtility - Output file created
ERROR 2019-07-03 03:06:10,043 [DAWSConnThread] App.DAErrorHandler - DAError: domainInitRqst::1::null::DDS_DATA_SOURCE_UNAVAILABLE::[[ERROR,Participant Error:null]]

input {
file{
path=> "c:/tmp/test.log"
start_position=>"beginning"
sincedb_path => "c:/tmp/null.sincedb"
}
}

filter {
grok{
match => { "message" => "%{LOGLEVEL:LEVEL} *%{TIMESTAMP_ISO8601:timestamp} [%{JAVACLASS:javaclass}] %{DATA:package}-%{DATA:action}\ *:%{GREEDYDATA:log}" }
}
}

output {
if "ERROR" in [LEVEL]
{file { path => "c:/tmp/test-error.txt" }}
file { path => "c:/tmp/test-error-2.txt" }
}

2

Your first problem is that your "JAVACLASS" does not match the JAVACLASS pattern, which requires two or more words separated using period.

I would use dissect to parse that.

dissect { mapping => { "message" => "%{LEVEL} %{timestamp} %{+timestamp} [%{javaclass}] %{package} - %{restOfLine}" } }

Then grok the [restOfLine] field.

If you want to go forward using grok make sure you understand why you should anchor your patterns to start of line if possible.

3

Thank you very much.