# Aggregate? **URL:** \ **Category:** Logstash\ **Created:** [April 15, 2016, 8:55am UTC](https://discuss.elastic.co/t/aggregate/47486 "2016-04-15T08:55:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Rubytor](https://avatars.discourse-cdn.com/v4/letter/r/5daacb/32.png) [@Rubytor](https://discuss.elastic.co/u/Rubytor)\ **Post date:** [April 15, 2016, 8:55am UTC](https://discuss.elastic.co/t/aggregate/47486/1 "2016-04-15T08:55:40Z")
What is the best solution for one of the basic requirements in log's analisis. "CALCULATE DURATION" The problem is to calculate the duration between 2 events. For example: Logs lines: 15-04-2016T10:00:00:000+UTC 1 start data 15-04-2016T10:00:00:001+UTC 2 start data 15-04-2016T10:00:00:004+UTC 2 end data 15-04-2016T10:00:00:005+UTC 1 end data In that case, it's necessary to add a field called "Duration" on the "end events" and assign the subtract the time. 15-04-2016T10:00:00:000+UTC 1 start data 0 15-04-2016T10:00:00:001+UTC 2 start data 0 15-04-2016T10:00:00:004+UTC 2 end data 3 15-04-2016T10:00:00:005+UTC 1 end data 5 Thank you to all. ---
**Author:** ![Gnosis](https://avatars.discourse-cdn.com/v4/letter/g/3e96dc/32.png) [@Gnosis](https://discuss.elastic.co/u/Gnosis)\ **Post date:** [April 15, 2016, 1:40pm UTC](https://discuss.elastic.co/t/aggregate/47486/2 "2016-04-15T13:40:22Z")
sorry i'm not sure to understand your question. ---
**Author:** ![Rubytor](https://avatars.discourse-cdn.com/v4/letter/r/5daacb/32.png) [@Rubytor](https://discuss.elastic.co/u/Rubytor)\ **Post date:** [April 16, 2016, 10:47am UTC](https://discuss.elastic.co/t/aggregate/47486/3 "2016-04-16T10:47:41Z")
Just calculate the time between 2 event. The events has an ID to correlate them and a flag like start, step1, setp2, end. I wanna know the duration from the start to step1, from start to step2, and total duration.. from start to end. Thank you ---
**Author:** ![jupp](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@jupp](https://discuss.elastic.co/u/jupp)\ **Post date:** [April 16, 2016, 7:38pm UTC](https://discuss.elastic.co/t/aggregate/47486/4 "2016-04-16T19:38:46Z")
Maybe this is the right filter : [https://www.elastic.co/guide/en/logstash/current/plugins-filters-elapsed.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elapsed.html) ---
**Author:** ![Rubytor](https://avatars.discourse-cdn.com/v4/letter/r/5daacb/32.png) [@Rubytor](https://discuss.elastic.co/u/Rubytor)\ **Post date:** [April 18, 2016, 7:27am UTC](https://discuss.elastic.co/t/aggregate/47486/5 "2016-04-18T07:27:18Z")
That's the solution if we want to calculte from the timestamp. The problem is that timestamp is assign by Logstash when an event comes. If we don't have a real time processing this isn't work. For that reason suppose an scenario like that. Logstash Processing Time ..... ID ........ TAG ........ Real time for the event 15-04-2016T10:00:00:000+UTC.... 1..... start ........ 15-04-2016T9:00:30:000+UTC 15-04-2016T10:00:00:005+UTC.... 1 ..... end ....... 15-04-2016T9:00:30:020+UTC If we used Elapsed plugin the duration will be 5 miliseconds, but the real duration should be calculate by Real Time field, in that case the duration will be 20 miliseconds. How can we do in that case??? ---
**Author:** ![Gnosis](https://avatars.discourse-cdn.com/v4/letter/g/3e96dc/32.png) [@Gnosis](https://discuss.elastic.co/u/Gnosis)\ **Post date:** [April 18, 2016, 9:17am UTC](https://discuss.elastic.co/t/aggregate/47486/6 "2016-04-18T09:17:36Z")
Hi, You should use the aggregate if you follow tis example [https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) [[https://www.elastic.co/static/img/elastic-logo-200.png](https://www.elastic.co/static/img/elastic-logo-200.png)][https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) aggregate - Elastic[https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) [www.elastic.co](http://www.elastic.co) The aim of this filter is to aggregate information available among several events (typically log lines) belonging to a same task, and finally push aggregated ... it should be fine by subtracting the time no ? ---
**Author:** ![jupp](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@jupp](https://discuss.elastic.co/u/jupp)\ **Post date:** [April 18, 2016, 6:47pm UTC](https://discuss.elastic.co/t/aggregate/47486/7 "2016-04-18T18:47:51Z")
@Rubytor You can overwrite the Logstash Processing Time with the real time for the event. Use the date filter for this purpose: > **[Date filter plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html)** @Gnosis When you use the aggregate-filter you must set filter workers to 1. This isn't really nice. > You should be very careful to set logstash filter workers to 1 (-w 1 flag) for this filter to work correctly otherwise documents may be processed out of sequence and unexpected results will occur. ---
**Author:** ![Gnosis](https://avatars.discourse-cdn.com/v4/letter/g/3e96dc/32.png) [@Gnosis](https://discuss.elastic.co/u/Gnosis)\ **Post date:** [April 19, 2016, 7:43am UTC](https://discuss.elastic.co/t/aggregate/47486/8 "2016-04-19T07:43:09Z")
Ok, thx a lot and how do you aggregate data without the aggregate-filter plz ? ---
**Author:** ![jupp](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@jupp](https://discuss.elastic.co/u/jupp)\ **Post date:** [April 19, 2016, 11:31am UTC](https://discuss.elastic.co/t/aggregate/47486/9 "2016-04-19T11:31:50Z")
> [@Gnosis](#): > > .. how do you aggregate data without the aggregate-filter plz ? Maybe your shipper can do this. I depends on your usecase and infrastructure. Image you have two logstash-instances with a load-balancer in front of them. How you would ensure that all Events flow to the same LS-Instance? ---
**Author:** ![Rubytor](https://avatars.discourse-cdn.com/v4/letter/r/5daacb/32.png) [@Rubytor](https://discuss.elastic.co/u/Rubytor)\ **Post date:** [April 19, 2016, 1:01pm UTC](https://discuss.elastic.co/t/aggregate/47486/10 "2016-04-19T13:01:17Z")
What about Filebeat?? I think it has it's own load balancer isn't it?? ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [April 24, 2016, 9:12pm UTC](https://discuss.elastic.co/t/aggregate/47486/11 "2016-04-24T21:12:48Z")
Up to me, the right solution for your need is to use 'date' filter and then 'elapsed' filter. - date filter allows you to put your message date (ex: 15-04-2016T10:00:00:000+UTC) in @timestamp field. - then elapsed filter will compute the elapsed time between start event and end event (using @timestamp field) and will store duration in 'elapsed.time' field in end event. But you have to know one thing : computed duration is in seconds. If you want a more precise duration (in milliseconds for example), you will have to use aggregate filter. In all cases, you must first use 'date' filter to set message date in @timestamp field. Here is the logstash configuration using aggregate filter : ``` date { match => ["_timestamp", "dd-MM-yyyy'T'HH:mm:ss:SSSZ"] } if [start] { aggregate { task_id => "%{taskid}" map_action => "create" code => "map['start_timestamp'] = event['@timestamp']" } } if [end] { aggregate { task_id => "%{taskid}" map_action => "update" code => "event['duration'] = event['@timestamp'] - map['start_timestamp']" end_of_task => true } } ``` Hope it helps. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:00am UTC](https://discuss.elastic.co/t/aggregate/47486/12 "2017-07-06T05:00:54Z")