_grokparsefailure Tag when parsing logs via logstash

I have the following log:-

2016-03-31 12:40:40 INFO SmartAppUtils:981 - Organization: AppsTwo
2016-03-31 12:40:40 INFO SmartAppUtils:988 - Brand organization: AppsTwo
2016-03-31 12:40:51 INFO SmartAppUtils:981 - Organization: AppsTwo
2016-03-31 12:40:51 INFO SmartAppUtils:988 - Brand organization: AppsTwo

My logstash configuration file as follows:-
input {
file {
type => "tomcat"
path => [ "D:/logs/Smart_logs/smartlogstest.log" ]
codec => multiline {
negate => true
pattern => "(^%{URIHOST} %{HAPROXYTIME})"
what => "previous"
}
}
}
filter {

if [type] == "tomcat" {
    
    grok{
		 patterns_dir => "./patterns"
         match => [ "message", "%{SMART_TIMESTAMP:timestamp} %{LOGLEVEL: logLevel}:%{GREEDYDATA:message}" ]
         overwrite => [ "message" ]
    }
	date{
		match=>["timestamp","yyyy-MM-dd HH:mm:ss"]
	}
   
}

}
output {
stdout { codec=>rubydebug }
elasticsearch{
hosts=>"localhost"
index=>"smartlogs_test"
}
}
The pattern under multiline codec is created using grok debugger, SMART_TIMESTAMP is a custom pattern which I have defined under patterns folder in a file named extra.conf
as follows:-
SMART_TIMESTAMP (%{URIHOST} %{HAPROXYTIME})

It seems that pattern is not matching but I have checked using grok debugger.
URIHOST returns 2016-03-31. and HAPROXYTIME returns 12:40:40

Comments:

  • Your definition of SMART_TIMESTAMP doesn't make sense. URIHOST has nothing to do with a yyyy-mm-dd date. Why not use TIMESTAMP_ISO8601 instead?
  • Remove the space between "LOGLEVEL:" and "logLevel".
  • In your grok expression you have a colon after the loglevel but there's no colon there in the actual log message.
1 Like

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

Thanks, magnusbaeck the logs are now started getting parsed by logstash. Could you please tell me that how we decide that we need to use TIMESTAMP_ISO8601. When I use grokdebugger why it didn't generated the pattern TIMESTAMP_ISO8601. May be it's a silly question but your answer will clear my doubts.

Below is my conf:-
input {
file {
type => "tomcat"
path => [ "D:/logs/Smart_logs/smartlogstest.log" ]
codec => multiline {
negate => true
pattern => "(^%{TIMESTAMP_ISO8601})"
what => "previous"
}
}
}
filter {

if [type] == "tomcat" {

grok{
     match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:logLevel}  %{NOTSPACE:className}:%{NUMBER:line} - %{GREEDYDATA:message}" ]
     overwrite => [ "message" ]
}
date{
	match=>["timestamp","yyyy-MM-dd HH:mm:ss"]
}

}
}
output {
stdout { codec=>rubydebug }
elasticsearch{
hosts=>"localhost"
index=>"smartlogs_test"
}
}

I have these log lines in the same log file as follows:-

2016-03-31 13:00:05 ERROR FlashSummarySubBuMismatchReportListener:47 - Scheduler failed ... Logging the error stack ... org.hibernate.exception.SQLGrammarException: could not execute query
2016-03-31 13:00:05 ERROR FlashSummarySubBuMismatchReportListener:48 - Scheduled job running failed ...org.hibernate.exception.SQLGrammarException: could not execute query
2016-03-31 13:00:07 ERROR EmailFactory:537 - Sending error information failed
2016-03-31 13:00:07 ERROR FlashSummaryLessFTEReportListener:54 - Scheduler failed ... Logging the error stack ... org.hibernate.exception.SQLGrammarException: could not execute query

They are not getting parsed. The pattern worked with earlier logs but not with these.

Could you please tell me that how we decide that we need to use TIMESTAMP_ISO8601. When I use grokdebugger why it didn't generated the pattern TIMESTAMP_ISO8601.

You mean grokconstructor? The grokdebugger site is only for testing your existing expression, right?

There can be many grok patterns that match a particular input string and a program doesn't always have the necessary context to make the right decision. Only you know what kind of data it is and what expression describes it best.

They are not getting parsed. The pattern worked with earlier logs but not with these.

In your expression you have two spaces between the LOGLEVEL and NOTSPACE patterns but your actual log doesn't have that.

1 Like

Thanks, for the reply, that was a silly mistake on my end.
The log:-
2016-03-31 12:40:51 INFO SmartAppUtils:988 - Brand organization: AppsTwo
2016-03-31 13:00:00 WARN FlashSummarySubBuMismatchReportListener:34 - Running FlashSummarySubBuMismatchReportListener ...
had two whitespaces between LOGLEVEL and NOTSPACE
but
the log:-
2016-03-31 13:00:05 ERROR FlashSummaryReportPrjBlankMessageListener:53 - Scheduler failed ... Logging the error stack ... org.hibernate.exception.SQLGrammarException: could not execute query
had one space between LOGLEVEL and NOTSPACE.

Because of this my expression was not working:-
%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:logLevel} %{NOTSPACE:className}:%{NUMBER:line} - %{GREEDYDATA:message}
as I assumed two whitespaces between LOGLEVEL and NOTSPACE.

Below is my corrected pattern:-
%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:logLevel}%{SPACE}%{NOTSPACE:className}:%{NUMBER:line} - %{GREEDYDATA:message}