# Aggregate filter with timeout\_timestamp\_field

**URL:** <https://discuss.elastic.co/t/aggregate-filter-with-timeout-timestamp-field/144260>\
**Category:** Logstash\
**Created:** [August 14, 2018, 12:15am UTC](https://discuss.elastic.co/t/aggregate-filter-with-timeout-timestamp-field/144260 "2018-08-14T00:15:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lakshmi-github](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshmi-github/32/81281_2.png) [@lakshmi-github](https://discuss.elastic.co/u/lakshmi-github)\
**Post date:** [August 14, 2018, 12:15am UTC](https://discuss.elastic.co/t/aggregate-filter-with-timeout-timestamp-field/144260/1 "2018-08-14T00:15:07Z")

</div>

Hello,

I am trying to get a count of certain log events aggregated by minute where the timestamp should be based on a date-time string in the log. Based on the documentation for aggregation, here is what I have so far.

Sample input log entries:

```
20180809-031301 Got a message
20180809-031331 Got a message
20180809-031401 Got a message
20180809-031431 Got a message

```

I would like logstash to show 2 messages each for minutes 20180809-0313 and 20180809-0314. This is US/Pacific time, in the logstash config below I convert these to UTC, save it to an event field and specify this field as the timeout\_timestamp\_field.

l.conf:

```
input {
  stdin { }
}

filter {
  grok {
    match => { "message" => "(?<logminute>^\d{8}-\d{4})\d{2}\s+Got a message.*" }
  }

  ruby {
   code => '
     require "tzinfo"
     t = Time.strptime(event.get("logminute") + "00", "%Y%m%d-%H%M%S")
     fromTZ = TZInfo::Timezone.get("US/Pacific")
     t_utc = fromTZ.local_to_utc(Time.new(t.year, t.month, t.day, t.hour, t.min, t.sec))
     event.set("logminute_utc", t_utc)
   '
  }

  aggregate {
    task_id => "%{logminute}"
    code => "map['cnt'] ||= 0; map['cnt'] += 1;"
    push_map_as_event_on_timeout => true
    timeout_tags => ['_aggregatetimeout']
    timeout => 60
    timeout_timestamp_field => "logminute_utc"
    timeout_code => '
      event.set("minute_rolled_over", "yes")
    '
  }

  mutate {
    remove_field => ["@timestamp", "host", "@version"]
  }
}

output {
  #if "_aggregatetimeout" in [tags] {
  stdout { codec => rubydebug }
  #}
}

```

I ran this with logstash-6.3.1 and only got the 4 events and no aggregation event.

```
{
    "logminute_utc" => 2018-08-09T10:14:00.000Z,
        "logminute" => "20180809-0314",
          "message" => "20180809-031401 Got a message"
}
{
    "logminute_utc" => 2018-08-09T10:14:00.000Z,
        "logminute" => "20180809-0314",
          "message" => "20180809-031431 Got a message"
}
{
    "logminute_utc" => 2018-08-09T10:13:00.000Z,
        "logminute" => "20180809-0313",
          "message" => "20180809-031301 Got a message"
}
{
    "logminute_utc" => 2018-08-09T10:13:00.000Z,
        "logminute" => "20180809-0313",
          "message" => "20180809-031331 Got a message"
}

```

Can you help with what I may be missing here.

Thanks in advance.

Lakshmi.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 14, 2018, 12:37am UTC](https://discuss.elastic.co/t/aggregate-filter-with-timeout-timestamp-field/144260/2 "2018-08-14T00:37:46Z")

</div>

Did you actually wait the 60 seconds for the timeouts to occur? Try it with a shorter timeout.

You probably want to add

```
timeout_task_id_field => "logminute"

```

I don't think timeout\_timestamp\_field does what you think it does, but I could be wrong on both of those "think" items 🙂

You may or may not want to add "event.cancel;" to the code option on the aggregate.

---

<div class="post-metadata">

**Author:** ![lakshmi-github](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshmi-github/32/81281_2.png) [@lakshmi-github](https://discuss.elastic.co/u/lakshmi-github)\
**Post date:** [August 14, 2018, 5:18am UTC](https://discuss.elastic.co/t/aggregate-filter-with-timeout-timestamp-field/144260/3 "2018-08-14T05:18:52Z")

</div>

I'm parsing old logs aggregated by 60 seconds worth of log entries as determined by log time stamp, so I figured I'd need the timeout\_timestamp\_field and 60 second timeout.

And for current logs also I am trying to link the aggregation interval to the log time stamp rather than system time, hence the need for timeout\_timestamp\_field.

At least this is my understanding from Elastic's documentation, [https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html)

Perhaps I'm missing something else.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 11, 2018, 5:19am UTC](https://discuss.elastic.co/t/aggregate-filter-with-timeout-timestamp-field/144260/4 "2018-09-11T05:19:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.