Failed Parsing Date and MulitLine tomcat log

Elastic Stack Logstash
#1

Hello, I've been trying to read tomcat log and make some analysis such as number of errors and exceptions raised etc. Here is my conf:[code]
input {
file {
path => "/opt/logstash/localhost.log"
start_position => beginning
sincedb_path => "/var/log/logstash/null"
}
}
filter {
grok {
match => { "message" => "(?%{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:logLevel} %{GREEDYDATA:message}" }
}
mutate {
add_field => { "logLevel" => "%{logLevel}" }
}
date {
locale => "en"
match => ["logdate", "dd-MMM-yyyy HH:mm:ss", "ISO8601"]
target => "@timestamp"
add_tag => ["tmatch"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}

[/code]
And below is some of the lines in my log file: 09-Jan-2016 18:30:38.722 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath 09-Jan-2016 18:30:38.796 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing log4j from [C:\tomcat\apache-tomcat-8.0.26\temp\0-contact-statecollab-ws-15.12-SNAPSHOT-unknown-20151230-1152\WEB-INF\log4j.properties

I have few of questions/problems:

  1. I got following exception:
    Failed parsing date from field {:field=>"logdate", :value=>"19-Jan-2016 18:30:38.722", :exception=>"Invalid format: \"19-Jan-2016 18:30:38.722\" is malformed at \"Jan-2016 18:30:38.722\"", :config_parsers=>"dd-MMM-yyyy HH:mm:ss,ISO8601", :config_locale=>"en", :level=>:warn}
  2. How should I use logdate instead of @timestamp
  3. How should I deal with multi line tomcat log?

Any suggestion would be appreciated. Thanks

#2

I found the problem with date, all I had to do was that add .SSS(second precision) to the date filter. So that it looks like:

date {
        locale => "en"
        match => ["logdate", "dd-MMM-yyyy HH:mm:ss.SSS", "ISO8601"]
    }

Now it gets timestamp correctly. But how should I deal with multiline error msg such as:

09-Jan-2016 18:43:29.405 SEVERE [http-apr-8080-exec-9] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
 org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'exports-ws': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: protected gov.hhs.acf.ohs.hses.model.exports.service.ExportsService gov.hhs.acf.ohs.hses.services.exports.ExportsWebService.exportsService; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [gov.hhs.acf.ohs.hses.model.exports.service.ExportsService] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true), @org.springframework.beans.factory.annotation.Qualifier(value=exportsService)}
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hses_service].
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactService' defined in class path resource [contactContext.xml]: Cannot resolve reference to bean 'contactPgTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgTransactionManager' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'contactPgSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgSessionFactory' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'gisDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hses_service].
#3

You need to use the multiline codec. Its documentation contains a few examples of how it can be used.

#4

Thank you Magnus, I added below lines to my config:

codec => multiline {
            pattern => "(dd-MMM-yyyy HH:mm:ss.SSS^\s*)"
            what => "next"
            }
        }

Basically, Each line should be started with date, if not, a line should be considered as continues of previous line. But it still count each line as a new line. Did I miss anything?

#5

Two problems:

  • Your pattern is not a regular expression. It looks like the pattern used for a date filter. Use your grok expression as a starting point instead.
  • The logic is flawed since it'll only work for sequences of two lines. If you have a sequence of three lines that should be joined, the second line won't begin with a timestamp so it won't be joined with the following line. Instead, turn the logic around with negate => true and use what => "previous", expressing "if the line doesn't begin with a timestamp, join it with the preceding line".
#6

Thank you Magnus, You explained it way better than the documentation. I replaced a pattern with grok expression and set max_lines since it's more than 500 rows. Now it's working as expected.

#7