Aggregate filter with timeout_timestamp_field

[lakshmi-github]

Hello,

I am trying to get a count of certain log events aggregated by minute where the timestamp should be based on a date-time string in the log. Based on the documentation for aggregation, here is what I have so far.

Sample input log entries:

20180809-031301 Got a message
20180809-031331 Got a message
20180809-031401 Got a message
20180809-031431 Got a message

I would like logstash to show 2 messages each for minutes 20180809-0313 and 20180809-0314. This is US/Pacific time, in the logstash config below I convert these to UTC, save it to an event field and specify this field as the timeout_timestamp_field.

l.conf:

input {
  stdin { }
}

filter {
  grok {
    match => { "message" => "(?<logminute>^\d{8}-\d{4})\d{2}\s+Got a message.*" }
  }

  ruby {
   code  => '
     require "tzinfo"
     t = Time.strptime(event.get("logminute") + "00", "%Y%m%d-%H%M%S")
     fromTZ = TZInfo::Timezone.get("US/Pacific")
     t_utc = fromTZ.local_to_utc(Time.new(t.year, t.month, t.day, t.hour, t.min, t.sec))
     event.set("logminute_utc", t_utc)
   '
  }

  aggregate {
    task_id => "%{logminute}"
    code => "map['cnt'] ||= 0; map['cnt'] += 1;"
    push_map_as_event_on_timeout => true
    timeout_tags => ['_aggregatetimeout']
    timeout => 60
    timeout_timestamp_field => "logminute_utc"
    timeout_code => '
      event.set("minute_rolled_over", "yes")
    '
  }

  mutate {
    remove_field => [ "@timestamp", "host", "@version" ]
  }
}

output {
  #if "_aggregatetimeout" in [tags] {
  stdout { codec => rubydebug }
  #}
}

I ran this with logstash-6.3.1 and only got the 4 events and no aggregation event.

{
    "logminute_utc" => 2018-08-09T10:14:00.000Z,
        "logminute" => "20180809-0314",
          "message" => "20180809-031401 Got a message"
}
{
    "logminute_utc" => 2018-08-09T10:14:00.000Z,
        "logminute" => "20180809-0314",
          "message" => "20180809-031431 Got a message"
}
{
    "logminute_utc" => 2018-08-09T10:13:00.000Z,
        "logminute" => "20180809-0313",
          "message" => "20180809-031301 Got a message"
}
{
    "logminute_utc" => 2018-08-09T10:13:00.000Z,
        "logminute" => "20180809-0313",
          "message" => "20180809-031331 Got a message"
}

Can you help with what I may be missing here.

Thanks in advance.

Lakshmi.

[Badger]

Did you actually wait the 60 seconds for the timeouts to occur? Try it with a shorter timeout.

You probably want to add

timeout_task_id_field => "logminute"

I don't think timeout_timestamp_field does what you think it does, but I could be wrong on both of those "think" items

You may or may not want to add "event.cancel;" to the code option on the aggregate.

[lakshmi-github]

I'm parsing old logs aggregated by 60 seconds worth of log entries as determined by log time stamp, so I figured I'd need the timeout_timestamp_field and 60 second timeout.

And for current logs also I am trying to link the aggregation interval to the log time stamp rather than system time, hence the need for timeout_timestamp_field.

At least this is my understanding from Elastic's documentation, https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html

Perhaps I'm missing something else.