Difference between timestamp

[Robert_Naccache]

Hi,

I've been struggling with this for a couple of days.
I'm importing from elasticsearch to elasticsearch.

In Elasticsearch i have data in the from of:

jobid: 1000
buisnessevent: getjob
alerteventtime: 2018-01-11T13:00:02.927Z

jobid: 1000
buisnessevent: getjob
alerteventtime: 2018-01-11T13:05:02.927Z

A single job id can have multiple documents of when the events occurred.

What i'm trying to do is find for every jobid, find the difference between the max time and the min time (duration).

I've tried the aggregate-plugin, but i can't seem to find a .max .min property to subtract them.
is the only way of doing this through ruby script? and if yes, do i actually have to implement the min/max feature or does ruby have its own (never used ruby)?

i'm kind of lost on which way i would go about solving this issue.

Thank you,
Robert

[Robert_Naccache]

Just a "where i'm at":

currently my aggregation filter looks like this:

aggregate {
	task_id => "%{jobid}"
	code => "
		map['max_value'] ||=0;
		map['min_value'] ||=0;
		map['time'] ||= [];
		map['time'] << {'eventtime' => event.get('alerteventtime')}
		map['jobid'] = event.get('jobid');
		map['businessevent'] = event.get('businessevent');
		event.cancel()
	"
	push_previous_map_as_event => true
}

which returns:

{
     "@version" => "1",
        "jobid" => 5773522705,
        "businessevent" => "HandleTrace",
         "time" => [
          [0] {"eventtime" => "2018-01-11T13:48:22.327Z"},
          [1] {"eventtime" => "2018-01-11T13:48:02.703Z"},
          [2] {"eventtime" => "2018-01-11T13:48:02.927Z"}
         ],
        "@timestamp" => 2018-01-12T10:40:58.710Z,
               "tags" => [
                        [0] "_aggregatefinalflush"
               ]
}

I'm almost there , any suggestions on how to get min and max time from "time" is highly appreciated!

[Robert_Naccache]

Figured it out!

I'll post my solution just in case someone is stuck trying to do something similar.

filter {

aggregate {
	task_id => "%{jobid}"
	code => "
		map['time'] ||= []; 
		map['time'] <<  event.get('alerteventtime');
		map['jobid'] = event.get('jobid');
		map['businessevent'] = event.get('businessevent');
		map['end'] = map['time'].max ;
		map['start'] = map['time'].min;
		event.cancel()
	"
	push_previous_map_as_event => true
}
mutate {
	remove_field => ["time"]
}

date {
	match => [ "start", "ISO8601" ]
	target => "start"
}
date {
	match => [ "end", "ISO8601" ]
	target => "end"
}
ruby {
	code => " 
		event.set('difference',  event.get('end') - event.get('start'));
	"
    }
}

I used the aggregate plugin to store the time values in an array and used .max .min to extract the max values (using event.cancel() to remove the default data). My time values were stored as strings so i needed the date plugin to format it correctly, and then used a ruby plugin to get the difference in seconds.

Don't know if this is the best way but it worked.

Thanks and have a nice day