Change the format of dates & calculate the difference

Elastic Stack Logstash
1

I am having a csv. In that I am having two times like this
Wed May 18 23:00:01.000 2016 | Wed May 18 23:01:32.236 2016

I want to change this as date format and find the difference between them. I tried like this. But while running ending up with errors but with configtest it says ok.

filter { csv {
columns => ["CDRtime","RcdNo","OrigNo","DestNo","MST","MDT","SrvTyp","DelvSts","Prio","RcdTyp"]
separator => "|" }
date { match => [ "MST", "UNIX" ] #request submittion time target => "MST" } }

Errors:

Failed parsing date from field {:field=>"MST", :value=>"Wed May 18 23:00:01 2016", :exception=>"Invalid UNIX epoch value 'Wed May 18 23:00:01 2016'", :config_parsers=>"UNIX", :config_locale=>"default=en_US", :level=>:warn}

2

Use a date format pattern that matches your input (the UNIX pattern isn't the right choice here). Consult the date filter documentation.

3

I checked all the patterns here https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns. But nothing suits me.

I tried like this now
match => [ "MST", "DAY MONTH ([1-9]{2}|\s[1-9]|10) TIME YEAR" ]

This also not working. How to match this

4

You're looking for a date pattern, not a grok pattern. See https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match and follow the joda.time.format.DateTimeFormat link.

1 Like
5

date { match => [ "MST", "EEE MMM dd HH:mm:ss.SSS YYYY", "EEE MMM d HH:mm:ss.SSS YYYY"] target => "MST" } date { match => [ "MDT", "EEE MMM dd HH:mm:ss.SSS YYYY", "EEE MMM d HH:mm:ss.SSS YYYY"] target => "MDT" }

Its solves my problem but I am seeing these fields as a string in Kibana. But I need that fields as timestamps.

For that how can I map the fields in ES. From this link
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html

But in my case how to write this mappings.

6

Elasticsearch should autodetect the MDT field as a date, but since you've previously indexed the field as a string that'll stick. The mapping of a field can't be changed without reindexing. But, as you presumably are using a time-series index this should correct itself until tomorrow.

To make sure a field is mapped a certain way you can modify the default index template used for your indexes.

1 Like