Tomcat/catalina.out logs are getting parsed by grok debugger but not by logstash

Elastic Stack Logstash
1

Hi,
We have tomcat/catalina.out log files which are getting parsed by grok debugger properly but when we provide same grok pattern in logstash indexer, it never parses it. In almost all logs it gives grokparsefailure.

In the pattern directory we have provided:

CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
DELIVERYTOMCAT %{CATALINA_DATESTAMP:timestamp}\s%{DATA:thread}\s*%{LOGLEVEL:severity}:\s%{DATA:message}\s:%{GREEDYDATA:query}

and sample log event is
Oct 29, 2016 11:03:09 PM org.mariadb.jdbc.internal.mysql.MySQLProtocol getResult
WARNING: Could not execute query sql : 'UPDATE tblJobStates js_forUpdate SET js_forUpdate.JobStatus = 'PROCESSING', js_forUpdate.Modifier = ?, js_forUpdate.DispatchId = ?, js_forUpdate.ModifiedTime = UTC_TIMESTAMP(), js_forUpdate.TimeoutTime = UTC_TIMESTAMP() + INTERVAL js_forUpdate.Timeout * 1000 MICROSECOND, js_forUpdate.RunCount = js_forUpdate.RunCount + 1 WHERE js_forUpdate.JobStatus = 'READY' ORDER BY js_forUpdate.Priority ASC, js_forUpdate.CreatedTime ASC LIMIT ?', parameters : ['google.com-pool-2-thread-4','40e0765a-0875-44534534534',10]: Deadlock found when trying to get lock; try restarting transaction

Thanks in advance
Ni3

2

Please show your grok filter and format the text as preformatted text using the toolbar button.

3

Here is our grok filter. We are using DELIVERYTOMCAT as the grok pattern.

CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
QUERY (?<query>(.|\r|\n)*)
DELIVERYTOMCAT %{CATALINA_DATESTAMP:timestamp}\s%{DATA:thread}\s*%{LOGLEVEL:severity}:\s%{DATA:message}\s:%{QUERY}

Thanks
Ni3

4

It's not immediately obvious what's wrong. General comments:

  • I'd be careful about those multiple DATA patterns. Unless you're very careful they might match things in unexpected ways.
  • In your example you're attempting to capture "org.mariadb.jdbc.internal.mysql.MySQLProtocol getResult" into the field thread, which doesn't seem to make sense. I'm sure "org.mariadb.jdbc.internal.mysql.MySQLProtocol" is the logger name and "getResult" looks like a method name. I suggest you use the NOTSPACE pattern to match these two tokens individually.
  • To debug things like this start with the simplest possible pattern, like %{MONTH}. If that works, add another token (%{MONTH} %{MONTHDAY}). Continue until things break. Then you've narrowed things down, hopefully enough to make it trivial to spot the problem.