Converting string to date

Elastic Stack Logstash
#1

Hi,

I am getting started with logstash and I am looking for some help.

I loaded the log file to ES but the logtimestamp field is a string. I want it to be a date. I understood how to convert to numeric data, but have not found anything for string-to-date conversion.

Sample data from the log file (parts were cut off to shorten the message):
Jul 1, 2015 5:15:04 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load ....
Jul 2, 2015 6:15:04 PM org.apache.catalina.loader.WebappClassLoader findResourceInternal
INFO: something bad: this web application instance has been stopped already. Could not load ...

I used mutliline first to merge the lines and then grok to parse the message field using patterns:

snippet from my config file:
grok {
patterns_dir => "my_grok_patterns"
match => { "message" => "%{CATALINA_DATESTAMP:logtimestamp} %{JAVACLASS:java_domain} %{WORD:java_class}%{GREEDYDATA:error_msg}" }
}

Thanks in advance for any pointers.

Frank.

How can I rename and reorder fields in Kibana?
#2

That's what the date filter is for.

Convert String to Date field within Kibana
#3

Thanks for the reply. I tried the following.

Dropped the index and related mapping.

Add a date filter:

date {
match => ["logtimestamp", "MMM dd, YYYY hh:mm:ss Z"]
}

The logtimestamp field is still defined as a string

Regards,
Frank

#4

If you want to have "logtimestamp" become a time object, you need to add target => "logtimestamp" to your date filter block.

By default, the date filter overwrites the @timestamp field with the value of the matched field, in this case, logtimestamp's value. Check to see if @timestamp matches the string value in "logtimestamp".

1 Like
#5

Hi Aaron. I got it to work. Thanks for your help!!

For completeness I added the steps below.

1. Multiline filter to combine the multi line messages into one line

multiline {
patterns_dir => "path_to_grok_patterns"
pattern => "(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})"
negate => true
what => "previous"
}

2. Parsed the message into separate fields using available patterns

grok {
patterns_dir => "path_to_grok_patterns"
match => { "message" => "%{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:java_class} %{WORD:java_method}%{GREEDYDATA:error_msg}" }
}

3. Replaced the @timestamp logstash field with the timestamp data from the file

date {
match => [ "timestamp", "MMM dd, yyyy HH:mm:ss a" ]
}

Regards,
Frank

1 Like
#6

Glad it worked out for you!

For bonus points, you can remove the now-superfluous timestamp field (since you overwrote @timestamp), by adding remove_field => "timestamp" to your date filter block. This will only delete the timestamp field upon successful conversion. This way you're not filling up your indices with a redundant timestamp field.

1 Like
#7

Sorry, for questioning on a closed thread.

date filter automatically converts the time from logs as per the time zone.
For example:
In log, if time stamp is '2016-09-15 09:40:24,118'
then output of date filter is '2016-09-15T04:10:24.118Z'

Please note that timezone is UTC+05:30.

I want to convert this string field to date for sorting purpose (that's why i am using date filter) and keep the same timestamp as it is in the logs .

Any help would be appreciated. :slight_smile:

#8

Solved the issue. Marking timezone as UTC (i.e. timezone => "UTC" ) would solve the above mentioned problem. But is it ok to manipulate settings in such a way (to get the desired output) ???

2 Likes
#9