Failed Parsing Date and MulitLine tomcat log

Hello, I've been trying to read tomcat log and make some analysis such as number of errors and exceptions raised etc. Here is my conf:[code]
input {
file {
path => "/opt/logstash/localhost.log"
start_position => beginning
sincedb_path => "/var/log/logstash/null"
}
}
filter {
grok {
match => { "message" => "(?%{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:logLevel} %{GREEDYDATA:message}" }
}
mutate {
add_field => { "logLevel" => "%{logLevel}" }
}
date {
locale => "en"
match => ["logdate", "dd-MMM-yyyy HH:mm:ss", "ISO8601"]
target => "@timestamp"
add_tag => ["tmatch"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}

[/code]
And below is some of the lines in my log file: 09-Jan-2016 18:30:38.722 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath 09-Jan-2016 18:30:38.796 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing log4j from [C:\tomcat\apache-tomcat-8.0.26\temp\0-contact-statecollab-ws-15.12-SNAPSHOT-unknown-20151230-1152\WEB-INF\log4j.properties

I have few of questions/problems:

  1. I got following exception:
    Failed parsing date from field {:field=>"logdate", :value=>"19-Jan-2016 18:30:38.722", :exception=>"Invalid format: \"19-Jan-2016 18:30:38.722\" is malformed at \"Jan-2016 18:30:38.722\"", :config_parsers=>"dd-MMM-yyyy HH:mm:ss,ISO8601", :config_locale=>"en", :level=>:warn}
  2. How should I use logdate instead of @timestamp
  3. How should I deal with multi line tomcat log?

Any suggestion would be appreciated. Thanks

I found the problem with date, all I had to do was that add .SSS(second precision) to the date filter. So that it looks like:

date {
        locale => "en"
        match => ["logdate", "dd-MMM-yyyy HH:mm:ss.SSS", "ISO8601"]
    }

Now it gets timestamp correctly. But how should I deal with multiline error msg such as:

09-Jan-2016 18:43:29.405 SEVERE [http-apr-8080-exec-9] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
 org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'exports-ws': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: protected gov.hhs.acf.ohs.hses.model.exports.service.ExportsService gov.hhs.acf.ohs.hses.services.exports.ExportsWebService.exportsService; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [gov.hhs.acf.ohs.hses.model.exports.service.ExportsService] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true), @org.springframework.beans.factory.annotation.Qualifier(value=exportsService)}
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hses_service].
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactService' defined in class path resource [contactContext.xml]: Cannot resolve reference to bean 'contactPgTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgTransactionManager' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'contactPgSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgSessionFactory' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'gisDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hses_service].

You need to use the multiline codec. Its documentation contains a few examples of how it can be used.

Thank you Magnus, I added below lines to my config:

codec => multiline {
            pattern => "(dd-MMM-yyyy HH:mm:ss.SSS^\s*)"
            what => "next"
            }
        }

Basically, Each line should be started with date, if not, a line should be considered as continues of previous line. But it still count each line as a new line. Did I miss anything?

Two problems:

  • Your pattern is not a regular expression. It looks like the pattern used for a date filter. Use your grok expression as a starting point instead.
  • The logic is flawed since it'll only work for sequences of two lines. If you have a sequence of three lines that should be joined, the second line won't begin with a timestamp so it won't be joined with the following line. Instead, turn the logic around with negate => true and use what => "previous", expressing "if the line doesn't begin with a timestamp, join it with the preceding line".

Thank you Magnus, You explained it way better than the documentation. I replaced a pattern with grok expression and set max_lines since it's more than 500 rows. Now it's working as expected.