# Clarify how timeout\_timestamp\_field works with timeouts in aggregate filter

**URL:** <https://discuss.elastic.co/t/clarify-how-timeout-timestamp-field-works-with-timeouts-in-aggregate-filter/178299>\
**Category:** Logstash\
**Created:** [April 24, 2019, 4:50pm UTC](https://discuss.elastic.co/t/clarify-how-timeout-timestamp-field-works-with-timeouts-in-aggregate-filter/178299 "2019-04-24T16:50:11Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![lens](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@lens](https://discuss.elastic.co/u/lens)\
**Post date:** [April 24, 2019, 4:50pm UTC](https://discuss.elastic.co/t/clarify-how-timeout-timestamp-field-works-with-timeouts-in-aggregate-filter/178299/1 "2019-04-24T16:50:11Z")

</div>

I am aggregating flows using timeouts based on a timestamp field in the events (start\_time), and I push\_map\_as\_event\_on\_timeout. Is my understanding of how this works correct?

An event comes in with a new task\_id (x), so a map is created for x having creation\_time = last\_start\_time = event's start\_time. When the next event with task\_id=x arrives, its start\_time is compared to last\_start\_time and creation\_time. If the differences are \< inactivity\_timeout and \< timeout, respectively, aggregate the 2 events using "code", update last\_start\_time in the map to the current event's start\_time, and keep going. However, if start\_time - last\_start\_time \> inactivity\_timeout OR if start\_time - creation\_time \> timeout, push the map as an event and start a new map; creation\_time and last\_start\_time are set to the new event's start\_time.

What happens if there are no more events with task\_id = x? How/when does the final map for x get flushed?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 24, 2019, 7:15pm UTC](https://discuss.elastic.co/t/clarify-how-timeout-timestamp-field-works-with-timeouts-in-aggregate-filter/178299/2 "2019-04-24T19:15:39Z")

</div>

My reading is that it [decides](https://github.com/logstash-plugins/logstash-filter-aggregate/blob/f3c2ee22d38510b9230eba753ed77e2e16458747/lib/logstash/filters/aggregate.rb#L191) whether to timeout the existing map element before it even looks at the current event.

flush() is called every 5 seconds. That [also](https://github.com/logstash-plugins/logstash-filter-aggregate/blob/f3c2ee22d38510b9230eba753ed77e2e16458747/lib/logstash/filters/aggregate.rb#L332) checks for expired maps.

---

<div class="post-metadata">

**Author:** ![lens](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@lens](https://discuss.elastic.co/u/lens)\
**Post date:** [April 24, 2019, 7:43pm UTC](https://discuss.elastic.co/t/clarify-how-timeout-timestamp-field-works-with-timeouts-in-aggregate-filter/178299/3 "2019-04-24T19:43:54Z")

</div>

That would make sense if you are not using timeout\_timestamp\_field (ie, using clock-time) but otherwise, it should use the specified field value in making the decision to expire/timeout. Right?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 24, 2019, 9:04pm UTC](https://discuss.elastic.co/t/clarify-how-timeout-timestamp-field-works-with-timeouts-in-aggregate-filter/178299/4 "2019-04-24T21:04:15Z")

</div>

> [@lens](#):
>
> it should use the specified field value in making the decision to expire/timeout. Right?

Yes, it does that. By storing the offset from the current time when creating a map entry it can add that back to the time on the event to get a time close to current time. Every 5 seconds it checks that last value to see if the delta from the current time is greater than the timeout interval.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2019, 9:04pm UTC](https://discuss.elastic.co/t/clarify-how-timeout-timestamp-field-works-with-timeouts-in-aggregate-filter/178299/5 "2019-05-22T21:04:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.