Timestamp difference calculation

Elastic Stack Logstash
1

Hello, I have a problem with the calculation of a time difference between two timestamps:

"time1" => "31/10/2017:10:18:19.928 +0200",
"time2" => "31/10/2017:10:19:00.190 +0200"

As a result I want to have a field in Logstash with the time difference.

The following code does not work:

grok {
break_on_match => false
match => [ "timerest", "%{SPACE}%{YEAR:year2}/%{MONTHNUM:month2}/%{MONTHDAY:day2}%{SPACE}%{HOUR:hour2}:%{MINUTE:minute2}:%{SECOND:second2}%{GREEDYDATA:timerest2}" ]
}
mutate {
add_field => { "time2" => "%{day2}/%{month2}/%{year2}:%{hour2}:%{minute2}:%{second2} +0200" }
}

date {
 match => [ "time2", "dd/MM/yyyy:HH:mm:ss.SSS Z" ]

}

mutate {
  remove_field => [ "year2", "month2", "day2", "hour2", "minute2", "second2", "timerest2" ]
}

date {
match => ["[Time1]", "ISO8601"]
target => "[Time1]"
}
date {
match => ["[Time2]", "ISO8601"]
target => "[Time2]"
}
ruby {
init => "require 'time'"
code => "duration = (event.get('time2') - event.get('time1')) rescue nil; event.set('Log_duration', duration); "
}

Thank you for your answers :slight_smile:

2

Field names are case-sentitive. You're storing to Time1 but reading from time1 in your ruby filter.

3

Hello,

Thank you, i changed the code:

mutate {
add_field => { "time1" => "%{day1}/%{month1}/%{year1}:%{hour1}:%{minute1}:%{second1} +0200" }
}

date {
  match => [ "time1", "yyyy-MM-dd HH:mm:ss.SSS Z" ]
}

mutate {
  remove_field => [ "year1", "month1", "day1", "hour1", "minute1", "second1" ]
}

grok {
break_on_match => false
match => [ "timerest", "%{SPACE}%{YEAR:year2}/%{MONTHNUM:month2}/%{MONTHDAY:day2}%{SPACE}%{HOUR:hour2}:%{MINUTE:minute2}:%{SECOND:second2}%{GREEDYDATA:timerest2}" ]
}
mutate {
add_field => { "time2" => "%{day2}/%{month2}/%{year2}:%{hour2}:%{minute2}:%{second2} +0200" }
}

date {
 match => [ "time2", "yyyy-MM-dd HH:mm:ss.SSS Z" ]

}

mutate {
  remove_field => [ "year2", "month2", "day2", "hour2", "minute2", "second2", "timerest2" ]
}

date {
match => ["[time1]", "ISO8601"]
target => "[time1]"
}

date {
match => ["[time2]", "ISO8601"]
target => "[time2]"
}
ruby {
init => "require 'time'"
code => "duration = (event.get('time2') - event.get('time1')) rescue nil; event.set('Log_duration', duration); "
}

}
}

Now i get the following error:
{:timestamp=>"2017-10-31T13:39:12.047000+0100", :message=>"Failed parsing date from field", :field=>"time1", :value=>"31/10/2017:13:38:42.259 +0200", :exception=>"Invalid format: "31/10/2017:13:38:42.259 +0200" is malformed at "/10/2017:13:38:42.259 +0200"", :config_parsers=>"yyyy-MM-dd HH:mm:ss.SSS Z", :config_locale=>"default=en_US", :level=>:warn}
{:timestamp=>"2017-10-31T13:39:12.053000+0100", :message=>"Failed parsing date from field", :field=>"time2", :value=>"31/10/2017:13:38:58.313 +0200", :exception=>"Invalid format: "31/10/2017:13:38:58.313 +0200" is malformed at "/10/2017:13:38:58.313 +0200"", :config_parsers=>"yyyy-MM-dd HH:mm:ss.SSS Z", :config_locale=>"default=en_US", :level=>:warn}
{:timestamp=>"2017-10-31T13:39:12.055000+0100", :message=>"Failed parsing date from field", :field=>"[time1]", :value=>"31/10/2017:13:38:42.259 +0200", :exception=>"Invalid format: "31/10/2017:13:38:42.259 +0200" is malformed at "/10/2017:13:38:42.259 +0200"", :config_parsers=>"ISO8601", :config_locale=>"default=en_US", :level=>:warn}
{:timestamp=>"2017-10-31T13:39:12.056000+0100", :message=>"Failed parsing date from field", :field=>"[time2]", :value=>"31/10/2017:13:38:58.313 +0200", :exception=>"Invalid format: "31/10/2017:13:38:58.313 +0200" is malformed at "/10/2017:13:38:58.313 +0200"", :config_parsers=>"ISO8601", :config_locale=>"default=en_US", :level=>:warn}

Any ideas?
Thank you,

4

The string "31/10/2017:13:38:42.259 +0200" clearly doesn't match the date pattern "yyyy-MM-dd HH:mm:ss.SSS Z".

5

Thank you for your help - now it is working fine :slight_smile:

Now i have the Duration in this Format:

"time1" => "2017-11-07T10:26:10.902Z",
"time2" => "2017-11-07T10:26:21.451Z",
"Log_duration" => 10.549

How can I transform this into minutes?

Thank you

6

How can I transform this into minutes?

I believe you need to use a ruby filter.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.