I am getting started with logstash and I am looking for some help.
I loaded the log file to ES but the logtimestamp field is a string. I want it to be a date. I understood how to convert to numeric data, but have not found anything for string-to-date conversion.
Sample data from the log file (parts were cut off to shorten the message):
Jul 1, 2015 5:15:04 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load ....
Jul 2, 2015 6:15:04 PM org.apache.catalina.loader.WebappClassLoader findResourceInternal
INFO: something bad: this web application instance has been stopped already. Could not load ...
I used mutliline first to merge the lines and then grok to parse the message field using patterns:
snippet from my config file:
grok {
patterns_dir => "my_grok_patterns"
match => { "message" => "%{CATALINA_DATESTAMP:logtimestamp} %{JAVACLASS:java_domain} %{WORD:java_class}%{GREEDYDATA:error_msg}" }
}
If you want to have "logtimestamp" become a time object, you need to add target => "logtimestamp" to your date filter block.
By default, the date filter overwrites the @timestamp field with the value of the matched field, in this case, logtimestamp's value. Check to see if @timestamp matches the string value in "logtimestamp".
For bonus points, you can remove the now-superfluous timestamp field (since you overwrote @timestamp), by adding remove_field => "timestamp" to your date filter block. This will only delete the timestamp field upon successful conversion. This way you're not filling up your indices with a redundant timestamp field.
date filter automatically converts the time from logs as per the time zone.
For example:
In log, if time stamp is '2016-09-15 09:40:24,118'
then output of date filter is '2016-09-15T04:10:24.118Z'
Please note that timezone is UTC+05:30.
I want to convert this string field to date for sorting purpose (that's why i am using date filter) and keep the same timestamp as it is in the logs .
Solved the issue. Marking timezone as UTC (i.e. timezone => "UTC" ) would solve the above mentioned problem. But is it ok to manipulate settings in such a way (to get the desired output) ???
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.