Hello, I've been trying to read tomcat log and make some analysis such as number of errors and exceptions raised etc. Here is my conf:[code]
input {
file {
path => "/opt/logstash/localhost.log"
start_position => beginning
sincedb_path => "/var/log/logstash/null"
}
}
filter {
grok {
match => { "message" => "(?%{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:logLevel} %{GREEDYDATA:message}" }
}
mutate {
add_field => { "logLevel" => "%{logLevel}" }
}
date {
locale => "en"
match => ["logdate", "dd-MMM-yyyy HH:mm:ss", "ISO8601"]
target => "@timestamp"
add_tag => ["tmatch"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}
[/code]
And below is some of the lines in my log file:
09-Jan-2016 18:30:38.722 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
09-Jan-2016 18:30:38.796 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing log4j from [C:\tomcat\apache-tomcat-8.0.26\temp\0-contact-statecollab-ws-15.12-SNAPSHOT-unknown-20151230-1152\WEB-INF\log4j.properties
I have few of questions/problems:
I got following exception: Failed parsing date from field {:field=>"logdate", :value=>"19-Jan-2016 18:30:38.722", :exception=>"Invalid format: \"19-Jan-2016 18:30:38.722\" is malformed at \"Jan-2016 18:30:38.722\"", :config_parsers=>"dd-MMM-yyyy HH:mm:ss,ISO8601", :config_locale=>"en", :level=>:warn}
I found the problem with date, all I had to do was that add .SSS(second precision) to the date filter. So that it looks like:
date {
locale => "en"
match => ["logdate", "dd-MMM-yyyy HH:mm:ss.SSS", "ISO8601"]
}
Now it gets timestamp correctly. But how should I deal with multiline error msg such as:
09-Jan-2016 18:43:29.405 SEVERE [http-apr-8080-exec-9] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'exports-ws': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: protected gov.hhs.acf.ohs.hses.model.exports.service.ExportsService gov.hhs.acf.ohs.hses.services.exports.ExportsWebService.exportsService; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [gov.hhs.acf.ohs.hses.model.exports.service.ExportsService] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true), @org.springframework.beans.factory.annotation.Qualifier(value=exportsService)}
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hses_service].
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactService' defined in class path resource [contactContext.xml]: Cannot resolve reference to bean 'contactPgTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgTransactionManager' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'contactPgSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgSessionFactory' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'gisDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hses_service].
Basically, Each line should be started with date, if not, a line should be considered as continues of previous line. But it still count each line as a new line. Did I miss anything?
Your pattern is not a regular expression. It looks like the pattern used for a date filter. Use your grok expression as a starting point instead.
The logic is flawed since it'll only work for sequences of two lines. If you have a sequence of three lines that should be joined, the second line won't begin with a timestamp so it won't be joined with the following line. Instead, turn the logic around with negate => true and use what => "previous", expressing "if the line doesn't begin with a timestamp, join it with the preceding line".
Thank you Magnus, You explained it way better than the documentation. I replaced a pattern with grok expression and set max_lines since it's more than 500 rows. Now it's working as expected.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.