Adding duration field from start date and end date

Elastic Stack Logstash
1

Hi. I am struggling to get the duration for a start and end date in a log entry. I have the following fields:
TIME=2016/05/29 13:23:04.286
TIME_END=2016/05/29 13:23:04.493

I would like to get the duration between the two fields, I have tried the following however it is not working:
mutate {
add_field => { elapsed => "%{TIME}-%{TIME_END}" }
}

and this code, however also did not work:
ruby {
code => "event['duration'] = (event['TIME']) - map['TIME_END']"
}

Any suggestions on how to get the duration between the two fields would be appreciated.

2

Use a ruby filter to convert those two fields to a timestamp value where date arithmetic works, then perform the subtraction. Something like

ruby {
  init => "require 'date'"
  code => "
    event['duration'] = 86400 * (
      DateTime.strptime(event['TIME_END'], '%Y/%m/%d %H:%M:%S.%L') - 
      DateTime.strptime(event['TIME'], '%Y/%m/%d %H:%M:%S.%L'))
  "
}

should work to get the duration in seconds.

3

Thank you very much for the response and the ruby script. I however can see currently different results:
203/1000
41/200
207/1000
51/250
211/1000
etc.

What do these results in the duration field exactly mean?

4

What, exactly, does your code look like?

5

Here is my code that I am using:
input {
file {
type => "probe"
path => [ "/Syslogs/_files/*.TXT" ]
start_position => beginning
sincedb_path => "/Syslogs/db/probe"
ignore_older => 864000000000
}
}

filter {
grok {
match => ["message", "%{GREEDYDATA:kv}"]
}
kv {
source => "kv"
field_split => ";"
value_split => "="
remove_field => "kv"
}
ruby {
init => "require 'date'"
code => "
event['duration'] = 86400 * (
DateTime.strptime(event['TIME_END'], '%Y/%m/%d %H:%M:%S.%L') -
DateTime.strptime(event['TIME'], '%Y/%m/%d %H:%M:%S.%L'))
"
}
date {
locale => "en"
match => ["TIME", "yyyy/MM/dd HH:mm:ss.SSS", "ISO8601"]
timezone => "Africa/Windhoek"
target => "@timestamp"
add_field => { "debug" => "timestampMatched"}
}
}

output {
elasticsearch {
hosts => "localhost:9200"
}

6

Just wanted to find out if anyone can help with this?

7

Hi @Hans and @magnusbaeck

in my log I have action start time and action end time formatted as HHmmss.
I used the following code in my logstash filter.
It works well if the action lasts during the same day.
However, I run into problems when the action end time passes midnight.
For example: this action's duration is 5 minutes
action start time 23:58:00
action end time 00:03:00 the next day
Then I get a wrong calculation (negative value).

I appreciate any help in solving this problem.

# get the duration of the action_time
     date {
            match => ["[action_start_time]", "HHmmss"]
            target => "[action_start_timed]"
            timezone => "America/New_York" 
       }
     date {
            match => ["[action_end_time]", "HHmmss"]
            target => "[action_end_timed]"
            timezone => "America/New_York" 
     }
     ruby {
            code => "event['action_duration'] = (event['action_end_timed'] - event['action_start_timed'])"
     }
8

@yahoo, please start your own thread.

1 Like